-- Logs begin at Tue 2022-07-26 01:47:35 UTC, end at Tue 2022-07-26 03:34:00 UTC. -- Jul 26 02:56:17.404542 dv-cinder-850840-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 26 02:56:17.413552 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 26 02:56:17.416396 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Jul 26 02:56:17 2022] *** Jul 26 02:56:17.416585 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 26 02:56:17.416728 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 26 02:56:17.416867 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: nodename: dv-cinder-850840-2-iscsi Jul 26 02:56:17.417003 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: machine: x86_64 Jul 26 02:56:17.417142 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: clock source: unix Jul 26 02:56:17.417267 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: pcre jit disabled Jul 26 02:56:17.417398 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: detected number of CPU cores: 4 Jul 26 02:56:17.417530 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: current working directory: / Jul 26 02:56:17.417661 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: detected binary path: /usr/bin/uwsgi-core Jul 26 02:56:17.417792 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: your processes number limit is 47680 Jul 26 02:56:17.417931 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: your memory page size is 4096 bytes Jul 26 02:56:17.418051 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: detected max file descriptor number: 2048 Jul 26 02:56:17.418180 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: lock engine: pthread robust mutexes Jul 26 02:56:17.418476 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: thunder lock: enabled Jul 26 02:56:17.418806 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 26 02:56:17.419043 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Jul 26 02:56:17.458678 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: Python main interpreter initialized at 0x5561bfbd49c0 Jul 26 02:56:17.458678 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: python threads support enabled Jul 26 02:56:17.458678 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: your server socket listen backlog is limited to 100 connections Jul 26 02:56:17.458678 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: your mercy for graceful operations on workers is 90 seconds Jul 26 02:56:17.459111 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: mapped 403077 bytes (393 KB) for 2 cores Jul 26 02:56:17.459111 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: *** Operational MODE: preforking *** Jul 26 02:56:17.459446 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: *** uWSGI is running in multiple interpreter mode *** Jul 26 02:56:17.459446 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: spawned uWSGI master process (pid: 80417) Jul 26 02:56:17.459680 dv-cinder-850840-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 26 02:56:17.460724 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: spawned uWSGI worker 1 (pid: 80427, cores: 1) Jul 26 02:56:17.460724 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: spawned uWSGI worker 2 (pid: 80428, cores: 1) Jul 26 02:56:17.460724 dv-cinder-850840-2-iscsi devstack@keystone.service[80417]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 26 02:56:22.945585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 26 02:56:22.945585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: from cryptography.utils import int_from_bytes Jul 26 02:56:22.946411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 26 02:56:22.946411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: from cryptography.utils import int_from_bytes Jul 26 02:56:22.946820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 26 02:56:22.946820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: from cryptography.utils import int_from_bytes Jul 26 02:56:22.946925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 26 02:56:22.946925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: from cryptography.utils import int_from_bytes Jul 26 02:56:22.997096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 26 02:56:22.997305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 26 02:56:22.997505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] command line args: [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 26 02:56:22.997759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 26 02:56:22.997871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 26 02:56:22.997937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ================================================================================ {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 26 02:56:22.998008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 26 02:56:22.998281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] admin_token = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:22.998375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] command line args: [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 26 02:56:22.998508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] config_dir = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:22.998585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 26 02:56:22.998714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ================================================================================ {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 26 02:56:22.998797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:22.998858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] config_source = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:22.998918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] admin_token = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:22.999124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] debug = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:22.999200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] config_dir = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:22.999270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] config_source = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] debug = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] list_limit = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] log_config_append = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.001949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] list_limit = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] log_dir = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] log_config_append = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] log_file = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] log_dir = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] log_file = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.002864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.003418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.003642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.003832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.004061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.004250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.004440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.004539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.004809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.005026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.005206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.005406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.005569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.005752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.005942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.006128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.006327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.006551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.006713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.006904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.006904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.007292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.007481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.007664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] publish_errors = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.007664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.008026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.008185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.008311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] publish_errors = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.008473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.008611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.008735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.008887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] standard_threads = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.009825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.010134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] standard_threads = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.010258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] use_journal = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.010387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.010527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] use_json = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.010652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.010786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] use_stderr = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.010938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.011103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] use_syslog = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.011228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] use_journal = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.011351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.011485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] use_json = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.011609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.011732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] use_stderr = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.011732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] use_syslog = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.011978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.012102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 26 02:56:23.012241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.012383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.012508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.012637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.012761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.015741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.015909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.016959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.017135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.017260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.017391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.019053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.019206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.019339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.019339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.019643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.019776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.019913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.020039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.020170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.020294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.020431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.020556 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.020761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.020888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.021928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.022919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.023988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.024111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.024235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.024363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.024495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.024619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.025114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.025807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.025938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026458 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.026993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.027121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.027244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.027366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.027488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.027691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.027818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.027942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.028958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.029080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.029220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.029355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.029570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.029696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.029819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.029942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.031372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.031566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.031758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.031944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.032129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.032311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.032500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.032686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.032866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.033047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.033229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.033471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.033666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.033825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.034020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.034204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.034204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.034554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.034554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.034894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.034894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.035284 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.035467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.035701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.035866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.036062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.036219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.036400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.036531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.036663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.036818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.036958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.037084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.037207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.037343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.037507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.037636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.037766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.037898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.038923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.039969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.040994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.041117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.041240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.041369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.041528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.041690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.041830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.041961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.042988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.043119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.043328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.043457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.043592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.043716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.043840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.043963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.044995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.045125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.045250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.045397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.045524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.045648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.045778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.045912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.046942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.047098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] auth.external = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.047233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.047428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.047576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.047704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.047831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.047955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.048079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.048214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] auth.password = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.048374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] auth.external = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.048510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] auth.token = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.048644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.048776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.048776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] auth.password = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] auth.token = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.049910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.050045 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.050178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.050304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.050435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] credential.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.050560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.050685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.050828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.051907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] credential.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.052931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.053056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.053188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.053188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.053424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.053664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.054855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.055122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.055250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.055382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.055508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.055632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.055765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.055900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.056928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.057939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.058063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.058208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.058336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.058468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.058592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.058716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.058841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.059965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.060091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.060222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.060348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.060481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.060607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.060796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.060922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061564 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.061826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.062070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.062195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.062320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.062445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.062570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.062694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.062826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.063799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.064914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.065946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.066994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.067136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.067263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.067388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.067521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.067654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.067781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.067905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.068922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069678 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.069937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.070981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.071112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.071237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.071378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.071506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.071631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.071756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.071975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.072131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.072268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.072407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.072533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.072657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.072782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.072914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.073922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.074980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.075121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.075255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.075389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.075515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.075640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.075764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.075888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.076952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.077952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.078981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.079122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.079250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.079374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.079374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.079620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.079743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.079875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.080905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.081922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.082931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.083131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.083258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] role.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.083382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.083506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] role.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.083630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.083827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.083955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.084971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] role.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] role.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.085982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.086113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.086239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.086364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.086488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.086626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.086760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.086886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.087919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.088936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.089949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.090971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.091102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.091228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.091364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.091491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.091626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.091750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.091968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.092104 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.092240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.092374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.092501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.092625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.092750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.092883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.093945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.094072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.094196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.094321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.094454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.094579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.094705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.094829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.095949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.096083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.096548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.096674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.096800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.096924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.connection = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.097962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.098996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.099134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.099271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.099397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.099522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.099647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.099780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.connection = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.099921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.100057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.100183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.100309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.100434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.100558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.100691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.101141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.101277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.101415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.101541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.101666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.101800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.101927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80428) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.102552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.103615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 26 02:56:23.103615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80427) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 26 02:56:23.110912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.111178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.111402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.112586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.112820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.113051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.119857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.121471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.149964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.150297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.151702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.151941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.206340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.206654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.257739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.257953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.258344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.258586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.322129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.322329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.322542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.322665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.322774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.322851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.322930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.323080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.323168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.323329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.323545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.323764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.346648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.346874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.347990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.348077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.348192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.348410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.348629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.359208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.359759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.372984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.373138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.373227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.373362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.373466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80428) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.373578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.373825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80427) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 26 02:56:23.390372 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.390550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 26 02:56:23.390721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.390891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.391032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 26 02:56:23.391221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 26 02:56:23.391347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.391431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 26 02:56:23.391505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.391735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.391876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.391961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.392036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.392222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.392349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.392455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.401848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.402119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.402243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.402370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.402463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.402668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.402796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.402914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.409607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.409924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.414428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.414726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.414890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.415032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.415113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.415295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.415422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.415547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.415639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.415825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.415966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.416051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.416127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.416339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.416466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.416580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.416683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.416876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.417004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.417111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.417200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.417403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.440343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.440586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.441860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.442135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.442287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.442370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.446765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.447067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.447349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.447556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.447843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.448169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.454101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.454383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.454894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.455234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.458581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.458826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.459127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.459399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.459548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.459679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.459807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.459934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.460060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.460187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.460331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.460459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.460589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.460840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.461112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.461387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.476838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.477126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.477861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.478167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.481073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.482149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.484329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.484624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.485405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.485701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.489030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.490154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.492227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.493372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.495415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.495713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.495991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.496279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.496563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.496648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.496854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.496940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.497145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.497438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.497725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.498017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.512976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.513269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.513521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.513851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.514254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.514550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.514818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.515109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.520798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.522056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.522352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.522756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.523696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.523987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.525463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.525824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.526020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.526332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.526617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.527966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.528252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.528542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.528837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.529125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.538159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.538445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.538726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.538987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.539301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.539575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.539840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.540723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.541019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.541303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.541586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.541868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.542144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.542432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.555508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.558240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.558486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.561310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.561485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.564200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.564366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.565730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.566012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.566285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.567010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.568547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.568837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.569117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.572816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.574417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.575643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.577433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.577557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.577721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.577999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.578268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.580426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.580715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.580998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.581274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.586337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.586623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.586899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.589372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.589656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.589940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.593839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80428) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.594118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.594396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.594666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.594969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.595233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.595502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.595775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.596143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.596787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.597074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.597267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80427) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 26 02:56:23.597350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.597551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.597636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.597824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80428) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.597916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.598103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.598387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.598670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.598941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.599247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.599627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.599913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.600207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.600473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.600749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.601010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80427) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 26 02:56:23.631214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x5561bfbd49c0 pid: 80428 (default app) Jul 26 02:56:23.634523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x5561bfbd49c0 pid: 80427 (default app) Jul 26 02:56:23.634716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ae464c-1984-4e02-a3c9-7b9f1397b2d5 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:23.635031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ae464c-1984-4e02-a3c9-7b9f1397b2d5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:23.635335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ae464c-1984-4e02-a3c9-7b9f1397b2d5 None None] PATH_INFO: `/v3` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:23.639687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1/1] 10.222.0.42 () {50 vars in 791 bytes} [Tue Jul 26 02:56:23 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:31.387197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2362140-5e7c-4812-8d8c-bb7274e8daa0 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:31.387787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2362140-5e7c-4812-8d8c-bb7274e8daa0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:31.387787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2362140-5e7c-4812-8d8c-bb7274e8daa0 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:31.392322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1/2] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:31 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:56:31.398100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:31.398571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:31.398660 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:31.471481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: INFO dbcounter [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] Registered counter for database keystone Jul 26 02:56:31.493266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG oslo_db.sqlalchemy.engines [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80428) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 26 02:56:31.499942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writer thread running {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 26 02:56:31.686059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG passlib.handlers.bcrypt [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80428) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 26 02:56:31.686398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG passlib.handlers.bcrypt [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80428) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 26 02:56:31.741811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:31.743492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:31.884219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 2/3] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:31 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 487 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 26 02:56:31.994968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2362140-5e7c-4812-8d8c-bb7274e8daa0 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:32.064573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: INFO dbcounter [None req-f2362140-5e7c-4812-8d8c-bb7274e8daa0 None None] Registered counter for database keystone Jul 26 02:56:32.084832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG oslo_db.sqlalchemy.engines [None req-f2362140-5e7c-4812-8d8c-bb7274e8daa0 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80427) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 26 02:56:32.092413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writer thread running {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 26 02:56:32.211728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2362140-5e7c-4812-8d8c-bb7274e8daa0 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:32.216472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeb0b56-8b9e-4961-abc9-8f09d51f1d15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:32.217184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeb0b56-8b9e-4961-abc9-8f09d51f1d15 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:32.217465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeb0b56-8b9e-4961-abc9-8f09d51f1d15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:32.217751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeb0b56-8b9e-4961-abc9-8f09d51f1d15 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:32.228606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeb0b56-8b9e-4961-abc9-8f09d51f1d15 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:32.280316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING keystone.common.rbac_enforcer.enforcer [None req-3aeb0b56-8b9e-4961-abc9-8f09d51f1d15 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 26 02:56:32.280531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeb0b56-8b9e-4961-abc9-8f09d51f1d15 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:32.291933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 2/4] 10.222.0.42 () {58 vars in 1324 bytes} [Tue Jul 26 02:56:31 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 02:56:32.298180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:32.313588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6838766-6248-48ff-90bb-f608cc5c1923 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:32.318582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35278a3-c399-40ad-8736-a6dd6be0bedf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:32.319796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35278a3-c399-40ad-8736-a6dd6be0bedf admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:32.320079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35278a3-c399-40ad-8736-a6dd6be0bedf admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:32.320346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35278a3-c399-40ad-8736-a6dd6be0bedf admin admin] PATH_INFO: `/v3/projects/b8c4cb2e7662425d87dcbcca96df5974` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:32.321923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35278a3-c399-40ad-8736-a6dd6be0bedf admin admin] RBAC: Authorizing `identity:get_project(project_id=b8c4cb2e7662425d87dcbcca96df5974)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:32.376609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f35278a3-c399-40ad-8736-a6dd6be0bedf admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 26 02:56:32.377220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35278a3-c399-40ad-8736-a6dd6be0bedf admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:32.383377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 3/5] 10.222.0.42 () {56 vars in 1207 bytes} [Tue Jul 26 02:56:32 2022] GET /identity/v3/projects/b8c4cb2e7662425d87dcbcca96df5974 => generated 344 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:34.389975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbe283d-28c0-4a9a-9bd8-bcaf47cc8702 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:34.390547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbe283d-28c0-4a9a-9bd8-bcaf47cc8702 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:34.390547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbe283d-28c0-4a9a-9bd8-bcaf47cc8702 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:34.394672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 3/6] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:34.400504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f0b03-8acb-4646-924c-ec4540bd05ff None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:34.400974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f0b03-8acb-4646-924c-ec4540bd05ff None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:34.401060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f0b03-8acb-4646-924c-ec4540bd05ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:34.421896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3e3f0b03-8acb-4646-924c-ec4540bd05ff None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:34.423495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3e3f0b03-8acb-4646-924c-ec4540bd05ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:34.432188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 4/7] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:34 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 26 02:56:34.454927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbe283d-28c0-4a9a-9bd8-bcaf47cc8702 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:34.468869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbe283d-28c0-4a9a-9bd8-bcaf47cc8702 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:34.475235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99f70f5-8077-4c7b-ae76-753d1eb7e92b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:34.475878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99f70f5-8077-4c7b-ae76-753d1eb7e92b admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:34.476271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99f70f5-8077-4c7b-ae76-753d1eb7e92b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:34.476569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99f70f5-8077-4c7b-ae76-753d1eb7e92b admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:34.486475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99f70f5-8077-4c7b-ae76-753d1eb7e92b admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:34.488330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99f70f5-8077-4c7b-ae76-753d1eb7e92b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:34.498601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 4/8] 10.222.0.42 () {58 vars in 1324 bytes} [Tue Jul 26 02:56:34 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 02:56:34.505140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3f0b03-8acb-4646-924c-ec4540bd05ff None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:34.518310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3f0b03-8acb-4646-924c-ec4540bd05ff None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:34.523443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456f938d-cab9-491c-bec6-ebc1e6df7251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:34.524089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456f938d-cab9-491c-bec6-ebc1e6df7251 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:34.524352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456f938d-cab9-491c-bec6-ebc1e6df7251 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:34.524670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456f938d-cab9-491c-bec6-ebc1e6df7251 admin admin] PATH_INFO: `/v3/users/84df6af7bcfb4cfca1045798a2ef6d69` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:34.526143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456f938d-cab9-491c-bec6-ebc1e6df7251 admin admin] RBAC: Authorizing `identity:get_user(user_id=84df6af7bcfb4cfca1045798a2ef6d69)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:34.528178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456f938d-cab9-491c-bec6-ebc1e6df7251 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:34.529802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 5/9] 10.222.0.42 () {56 vars in 1198 bytes} [Tue Jul 26 02:56:34 2022] GET /identity/v3/users/84df6af7bcfb4cfca1045798a2ef6d69 => generated 247 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:36.601682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a417a6a-a060-411c-a855-905f74eca5ad None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:36.602708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a417a6a-a060-411c-a855-905f74eca5ad None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:36.603250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a417a6a-a060-411c-a855-905f74eca5ad None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:36.606087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 5/10] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:36.610816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6e2aa0-1f69-4a4b-9c8a-fa3de1d91ef2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:36.611264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6e2aa0-1f69-4a4b-9c8a-fa3de1d91ef2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:36.611432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6e2aa0-1f69-4a4b-9c8a-fa3de1d91ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:36.625109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371ccb0f-970b-4983-815c-20fab402ce85 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:36.625553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371ccb0f-970b-4983-815c-20fab402ce85 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:36.625705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371ccb0f-970b-4983-815c-20fab402ce85 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:36.627052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 6/11] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:36.632683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:36.632969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:36.633242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:36.633525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4c6e2aa0-1f69-4a4b-9c8a-fa3de1d91ef2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:36.634829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4c6e2aa0-1f69-4a4b-9c8a-fa3de1d91ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:36.680704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG passlib.handlers.bcrypt [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80427) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 26 02:56:36.681143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG passlib.handlers.bcrypt [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80427) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 26 02:56:36.720562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 6/12] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:56:36.726700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:36.728358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:36.743516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6e2aa0-1f69-4a4b-9c8a-fa3de1d91ef2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:36.768367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:36.832030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 7/13] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:36.853092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6e2aa0-1f69-4a4b-9c8a-fa3de1d91ef2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:36.854157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:36.886543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:36.887221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:36.887491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:36.887761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:36.899127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:36.907995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:36.908906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:36.909872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 7/14] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:56:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:56:36.915152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:36.959688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6614f135-a946-40d8-be15-359aa5c96e48 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:36.962019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:36.997192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:36.998977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:37.000029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:37.000303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:37.000548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:37.015762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195de1c8-6ae8-4232-8425-75fbe207e856 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:37.049300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:37.049933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:37.050220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:37.050495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:37.051323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:37.051634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:37.053059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:37.064814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 8/15] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:37.070492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:37.071721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:37.073256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:37.169912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c513de7-31f0-4703-a216-963f9aa5f284 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:37.172146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 8/16] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:56:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:56:37.177851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:37.181110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:37.289254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debc87e4-f176-4690-bdec-02d252396546 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:37.317297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:37.317960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:37.318235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:37.318508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] PATH_INFO: `/v3/users/84df6af7bcfb4cfca1045798a2ef6d69` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:37.319799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] RBAC: Authorizing `identity:get_user(user_id=84df6af7bcfb4cfca1045798a2ef6d69)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:37.321276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:37.322880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 9/17] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:37 2022] GET /identity/v3/users/84df6af7bcfb4cfca1045798a2ef6d69 => generated 247 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:37.326371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8423314d-f626-4638-a286-ab6f055967be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:37.327473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8423314d-f626-4638-a286-ab6f055967be None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:37.327753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8423314d-f626-4638-a286-ab6f055967be None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:37.328029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8423314d-f626-4638-a286-ab6f055967be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:37.328516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:37.343695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:37.347919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8423314d-f626-4638-a286-ab6f055967be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:37.349430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8423314d-f626-4638-a286-ab6f055967be None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-8423314d-f626-4638-a286-ab6f055967be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:56:37.368546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:37.369518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:37.370301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:56:37.370301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:56:37.370301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:56:37.370301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:56:37.370301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 9/18] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:56:37.376000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8423314d-f626-4638-a286-ab6f055967be None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:37.430386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc94218-1210-4cca-8d4c-a9ab04528496 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:37.431706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:37.465879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:37.466586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:37.466883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:37.467227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:37.468449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:37.469972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:37.471989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 10/19] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:37.477373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8423314d-f626-4638-a286-ab6f055967be None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:37.478689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:37.511281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73a6001-d462-49b1-aca7-82ebd7a09879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:37.511915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a6001-d462-49b1-aca7-82ebd7a09879 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:37.512195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a6001-d462-49b1-aca7-82ebd7a09879 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:37.512467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73a6001-d462-49b1-aca7-82ebd7a09879 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:37.513178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73a6001-d462-49b1-aca7-82ebd7a09879 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:37.513563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73a6001-d462-49b1-aca7-82ebd7a09879 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:37.514991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73a6001-d462-49b1-aca7-82ebd7a09879 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:37.532533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 10/20] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:56:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:37.562043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:37.586199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c81e6-53e7-452a-a283-80fb4be6c887 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:37.626726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357206ed-dbd9-470f-a6e7-6dbaa35d83b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:37.627471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357206ed-dbd9-470f-a6e7-6dbaa35d83b0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:37.627864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357206ed-dbd9-470f-a6e7-6dbaa35d83b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:37.628193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357206ed-dbd9-470f-a6e7-6dbaa35d83b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:37.628967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357206ed-dbd9-470f-a6e7-6dbaa35d83b0 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.domain.id=default, user.id=84df6af7bcfb4cfca1045798a2ef6d69) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:37.629284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357206ed-dbd9-470f-a6e7-6dbaa35d83b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:37.630779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357206ed-dbd9-470f-a6e7-6dbaa35d83b0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:37.639237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 11/21] 10.222.0.42 () {56 vars in 1322 bytes} [Tue Jul 26 02:56:37 2022] GET /identity/v3/role_assignments?user.id=84df6af7bcfb4cfca1045798a2ef6d69&scope.domain.id=default&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 233 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:39.801506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6cdfd-d019-4b4e-9107-e5bdee464311 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:39.802388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6cdfd-d019-4b4e-9107-e5bdee464311 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:39.802816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6cdfd-d019-4b4e-9107-e5bdee464311 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:39.804559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 11/22] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:39.811331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceeeb5b-d302-433c-9845-51fe1da7b250 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:39.812167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceeeb5b-d302-433c-9845-51fe1da7b250 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:39.812582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceeeb5b-d302-433c-9845-51fe1da7b250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:39.835325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9ceeeb5b-d302-433c-9845-51fe1da7b250 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:39.836778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9ceeeb5b-d302-433c-9845-51fe1da7b250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:39.928992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 12/23] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:39.963798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df6cdfd-d019-4b4e-9107-e5bdee464311 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:40.058869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21789aa-6753-4a97-b39c-e2d4361cd0c0 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.059503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21789aa-6753-4a97-b39c-e2d4361cd0c0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.059999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21789aa-6753-4a97-b39c-e2d4361cd0c0 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.061635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 13/24] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:40.067748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15b285-8370-4869-95a5-65ff413e0907 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.068273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15b285-8370-4869-95a5-65ff413e0907 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.068757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15b285-8370-4869-95a5-65ff413e0907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.096660 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ce15b285-8370-4869-95a5-65ff413e0907 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:40.098168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ce15b285-8370-4869-95a5-65ff413e0907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:40.106967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df6cdfd-d019-4b4e-9107-e5bdee464311 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:40.132772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:40.164588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:40.165249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.165527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.165798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.181307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:40.182838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:56:40.213893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:40.214758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:40.215701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:56:40.215701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:56:40.215701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:56:40.215701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:56:40.215701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 12/25] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:40.223189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:40.248936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 14/26] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:56:40.289414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce15b285-8370-4869-95a5-65ff413e0907 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:40.299599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:40.431552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28207c90-bd6e-4ee1-be09-f38cf1f391cc None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:40.489263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce15b285-8370-4869-95a5-65ff413e0907 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:40.496215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b60cc0b-229e-4adb-88a2-17fa48b16976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:40.496858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b60cc0b-229e-4adb-88a2-17fa48b16976 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.497260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b60cc0b-229e-4adb-88a2-17fa48b16976 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.497537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b60cc0b-229e-4adb-88a2-17fa48b16976 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.498278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b60cc0b-229e-4adb-88a2-17fa48b16976 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:40.498582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b60cc0b-229e-4adb-88a2-17fa48b16976 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:40.500106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b60cc0b-229e-4adb-88a2-17fa48b16976 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:40.520027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 13/27] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:56:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:40.521515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de52f14-9159-4c0e-9410-6b4e732b17c9 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.521800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de52f14-9159-4c0e-9410-6b4e732b17c9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.522076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de52f14-9159-4c0e-9410-6b4e732b17c9 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.523612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 14/28] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:40.527333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6dabe8-0842-4b66-8401-cc2e1f133909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:40.528266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de52f14-9159-4c0e-9410-6b4e732b17c9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:40.528763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6dabe8-0842-4b66-8401-cc2e1f133909 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.529813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6dabe8-0842-4b66-8401-cc2e1f133909 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.530309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6dabe8-0842-4b66-8401-cc2e1f133909 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.531275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6dabe8-0842-4b66-8401-cc2e1f133909 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:40.532903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6dabe8-0842-4b66-8401-cc2e1f133909 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:40.664873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 15/29] 10.222.0.42 () {60 vars in 1131 bytes} [Tue Jul 26 02:56:40 2022] POST /identity/v3/roles => generated 219 bytes in 377 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:56:40.666910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd81dbc6-a94d-46cd-ab38-431db2ef769f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.667457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd81dbc6-a94d-46cd-ab38-431db2ef769f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.668032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd81dbc6-a94d-46cd-ab38-431db2ef769f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.689541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-cd81dbc6-a94d-46cd-ab38-431db2ef769f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:40.691573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cd81dbc6-a94d-46cd-ab38-431db2ef769f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:40.696514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:40.767495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:40.774200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 16/30] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:40.776271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885239bc-41be-404a-8d8c-185f0eeae927 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.777056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885239bc-41be-404a-8d8c-185f0eeae927 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.777527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885239bc-41be-404a-8d8c-185f0eeae927 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.779134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 17/31] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:40.795179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf8e2a0-dcb2-40b0-9a03-ed4f60290487 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.795832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf8e2a0-dcb2-40b0-9a03-ed4f60290487 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.796357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf8e2a0-dcb2-40b0-9a03-ed4f60290487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.836886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ddf8e2a0-dcb2-40b0-9a03-ed4f60290487 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:40.838137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ddf8e2a0-dcb2-40b0-9a03-ed4f60290487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:40.869567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de52f14-9159-4c0e-9410-6b4e732b17c9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:40.925613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:40.927219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:40.927505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:40.927826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:40.928803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:40.930911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:40.946873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 18/32] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:56:40.947724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf8e2a0-dcb2-40b0-9a03-ed4f60290487 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.077935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf8e2a0-dcb2-40b0-9a03-ed4f60290487 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.108407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 15/33] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:56:40 2022] POST /identity/v3/projects => generated 301 bytes in 580 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:56:41.109795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.110229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:41.125929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.126701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.127094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.127529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.140226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:41.145784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:41.147673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:41.148665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 19/34] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:56:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:41.156286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.219134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:41.277463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cc02b8-978a-4e2e-9840-e9c47582caec None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.319188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df656b9-c0dc-4c91-8336-ece26dd83be1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.363293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40e418e-aea0-4e7d-9bd7-9f83f4e1bf2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.363636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e683ea-9059-49e6-bca2-2ff3f0940fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.363956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40e418e-aea0-4e7d-9bd7-9f83f4e1bf2a None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.364139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e683ea-9059-49e6-bca2-2ff3f0940fef None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.364281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40e418e-aea0-4e7d-9bd7-9f83f4e1bf2a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.364420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e683ea-9059-49e6-bca2-2ff3f0940fef None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.364558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40e418e-aea0-4e7d-9bd7-9f83f4e1bf2a None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.364704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e683ea-9059-49e6-bca2-2ff3f0940fef None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.365229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40e418e-aea0-4e7d-9bd7-9f83f4e1bf2a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:41.365395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e683ea-9059-49e6-bca2-2ff3f0940fef None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:41.365700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e683ea-9059-49e6-bca2-2ff3f0940fef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:41.366783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40e418e-aea0-4e7d-9bd7-9f83f4e1bf2a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:41.367293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e683ea-9059-49e6-bca2-2ff3f0940fef None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:41.378408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 20/35] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:41.382291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c476b4c-32c9-4e21-8fae-80dbd7ea07e4 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.382663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c476b4c-32c9-4e21-8fae-80dbd7ea07e4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.383057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c476b4c-32c9-4e21-8fae-80dbd7ea07e4 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.385090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 21/36] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:41.388021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c476b4c-32c9-4e21-8fae-80dbd7ea07e4 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.391669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 16/37] 10.222.0.42 () {60 vars in 1131 bytes} [Tue Jul 26 02:56:41 2022] POST /identity/v3/roles => generated 225 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:56:41.393189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44224b1f-6938-4621-ab6a-f431d71959d4 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.393482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44224b1f-6938-4621-ab6a-f431d71959d4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.393760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44224b1f-6938-4621-ab6a-f431d71959d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.414989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-44224b1f-6938-4621-ab6a-f431d71959d4 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:41.416887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-44224b1f-6938-4621-ab6a-f431d71959d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:41.454141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:41.510885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 17/38] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:41.512190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e687f205-aa27-4594-8d9e-a50745a98d3e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.512477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e687f205-aa27-4594-8d9e-a50745a98d3e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.512755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e687f205-aa27-4594-8d9e-a50745a98d3e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.514121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 18/39] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:41.522401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f5359-6712-4138-9da3-308f2e5e8232 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.522787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f5359-6712-4138-9da3-308f2e5e8232 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.523168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f5359-6712-4138-9da3-308f2e5e8232 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.541592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c476b4c-32c9-4e21-8fae-80dbd7ea07e4 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.551054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-fb0f5359-6712-4138-9da3-308f2e5e8232 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:41.552429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fb0f5359-6712-4138-9da3-308f2e5e8232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:41.584540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.585292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.585644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.585931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] PATH_INFO: `/v3/users/84df6af7bcfb4cfca1045798a2ef6d69` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.587197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] RBAC: Authorizing `identity:get_user(user_id=84df6af7bcfb4cfca1045798a2ef6d69)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:41.588705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:41.590349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 22/40] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:41 2022] GET /identity/v3/users/84df6af7bcfb4cfca1045798a2ef6d69 => generated 247 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:41.591322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.638752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:41.652761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 19/41] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:41.653893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0f5359-6712-4138-9da3-308f2e5e8232 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.707607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4559b122-b3c1-4db6-bdd4-b0b2a1012cbf None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.721304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:41.742940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.743630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.743918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.744197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.745407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:41.746896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:41.748691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 23/42] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:41.749455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.754387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0f5359-6712-4138-9da3-308f2e5e8232 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.787752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.788463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.788816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.789099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.790278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:41.791800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:41.793395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 20/43] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:41.794179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.854099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836d164-3f22-4e02-90a4-a4a3a9c6cec5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.871585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:41.888080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.888784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.889070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.889350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.890082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:41.891543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:41.906071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 24/44] 10.222.0.42 () {60 vars in 1131 bytes} [Tue Jul 26 02:56:41 2022] POST /identity/v3/roles => generated 223 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:56:41.906444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c46679-8046-4719-800c-ebe14f49d12d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:41.906853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:41.956869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4b9bc3-e170-47a6-8666-d93a96f89a8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:41.957517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b9bc3-e170-47a6-8666-d93a96f89a8d None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:41.957815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b9bc3-e170-47a6-8666-d93a96f89a8d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:41.958094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b9bc3-e170-47a6-8666-d93a96f89a8d None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:41.958817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4b9bc3-e170-47a6-8666-d93a96f89a8d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:41.960377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4b9bc3-e170-47a6-8666-d93a96f89a8d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:41.965807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:41.992771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 21/45] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:56:41 2022] POST /identity/v3/projects => generated 312 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:56:42.043037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1a3672-e4ba-4f85-afb2-eec64ad92e0e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:42.087866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f561e-0457-4358-8d52-899e89376ef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:42.088586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f561e-0457-4358-8d52-899e89376ef4 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:42.088877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f561e-0457-4358-8d52-899e89376ef4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:42.089162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f561e-0457-4358-8d52-899e89376ef4 None admin] PATH_INFO: `/v3/domains/default/users/84df6af7bcfb4cfca1045798a2ef6d69/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:42.091325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18f561e-0457-4358-8d52-899e89376ef4 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=84df6af7bcfb4cfca1045798a2ef6d69, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:42.093253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18f561e-0457-4358-8d52-899e89376ef4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:42.110110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 25/46] 10.222.0.42 () {58 vars in 1361 bytes} [Tue Jul 26 02:56:41 2022] PUT /identity/v3/domains/default/users/84df6af7bcfb4cfca1045798a2ef6d69/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:56:44.022423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aabbcb-fceb-41a7-ba3a-c0717c2d9708 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.022862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aabbcb-fceb-41a7-ba3a-c0717c2d9708 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.023069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aabbcb-fceb-41a7-ba3a-c0717c2d9708 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.024422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 22/47] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:44.029988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c03ef9-71b0-4064-84da-361f73382d59 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.030404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c03ef9-71b0-4064-84da-361f73382d59 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.030549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c03ef9-71b0-4064-84da-361f73382d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.050618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-20c03ef9-71b0-4064-84da-361f73382d59 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:44.051879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-20c03ef9-71b0-4064-84da-361f73382d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:44.081190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdc8c96-ee5c-40e0-a728-1232d9e6ab4b None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.081481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdc8c96-ee5c-40e0-a728-1232d9e6ab4b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.081844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdc8c96-ee5c-40e0-a728-1232d9e6ab4b None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.083972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 23/48] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:56:44.089670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94923d67-2509-4d18-95f5-3c64fc722e17 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.090035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94923d67-2509-4d18-95f5-3c64fc722e17 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.090423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94923d67-2509-4d18-95f5-3c64fc722e17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.109386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-94923d67-2509-4d18-95f5-3c64fc722e17 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:44.110606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-94923d67-2509-4d18-95f5-3c64fc722e17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:44.126241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 26/49] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:44.149632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c03ef9-71b0-4064-84da-361f73382d59 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:44.181347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:44.181527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 24/50] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:44.206080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94923d67-2509-4d18-95f5-3c64fc722e17 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:44.254226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c03ef9-71b0-4064-84da-361f73382d59 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:44.257220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:44.290428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:44.291054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.291339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.291623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.292813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:44.294280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:44.295959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 27/51] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:44.301630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:44.307581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94923d67-2509-4d18-95f5-3c64fc722e17 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:44.344310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:44.344960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.345248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.345507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.357861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:44.364981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:44.365866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:56:44.366624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 25/52] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:56:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:56:44.372211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:44.414235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:44.418931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4455614-7f3a-4273-99d9-8cf90a1fa1f6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:44.454523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:44.455226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.455474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.455755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.456759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:44.458292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:44.489083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 28/53] 10.222.0.42 () {60 vars in 1140 bytes} [Tue Jul 26 02:56:44 2022] POST /identity/v3/projects => generated 298 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:56:44.490652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fab6d3-a1da-448d-b930-506ee33186e9 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:44.502611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:44.529624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:44.530273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.530559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.530826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.531629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:44.531943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:44.533388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:44.547867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 26/54] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:44.554166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:44.675971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ccee49-b136-433b-b352-f992e0714b8a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:44.718383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:44.719078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.719363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.719642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] PATH_INFO: `/v3/users/84df6af7bcfb4cfca1045798a2ef6d69` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.721242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] RBAC: Authorizing `identity:get_user(user_id=84df6af7bcfb4cfca1045798a2ef6d69)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:44.722724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:44.724403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 29/55] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:44 2022] GET /identity/v3/users/84df6af7bcfb4cfca1045798a2ef6d69 => generated 247 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:44.730168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:44.834413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdcabe2-dad0-4e1b-93fd-5e608dafb507 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:44.870616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fe7b4a-0fa1-422c-aca3-114b37243708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:44.871293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe7b4a-0fa1-422c-aca3-114b37243708 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:44.871583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe7b4a-0fa1-422c-aca3-114b37243708 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:44.871859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe7b4a-0fa1-422c-aca3-114b37243708 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:44.873118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fe7b4a-0fa1-422c-aca3-114b37243708 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:44.874595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fe7b4a-0fa1-422c-aca3-114b37243708 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:44.876487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 27/56] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:44.882224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:44.905830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:44.993747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2195039-95d8-4c63-8662-b7f713ea6c82 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:45.028820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e1369-0941-417b-bc5a-560fb36418e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:45.029549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e1369-0941-417b-bc5a-560fb36418e5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:45.029832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e1369-0941-417b-bc5a-560fb36418e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:45.030109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e1369-0941-417b-bc5a-560fb36418e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:45.030906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7e1369-0941-417b-bc5a-560fb36418e5 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.domain.id=default, user.id=84df6af7bcfb4cfca1045798a2ef6d69) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:45.031247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7e1369-0941-417b-bc5a-560fb36418e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:45.032706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7e1369-0941-417b-bc5a-560fb36418e5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:45.038595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 30/57] 10.222.0.42 () {56 vars in 1322 bytes} [Tue Jul 26 02:56:44 2022] GET /identity/v3/role_assignments?user.id=84df6af7bcfb4cfca1045798a2ef6d69&scope.domain.id=default&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 530 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:46.498587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb49ca-9d93-4eed-8834-29a470ec228e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:46.499155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb49ca-9d93-4eed-8834-29a470ec228e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:46.499155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb49ca-9d93-4eed-8834-29a470ec228e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:46.501202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 28/58] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:46.506548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da56638b-777b-4fe2-ba35-bd753057a3d6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:46.506981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da56638b-777b-4fe2-ba35-bd753057a3d6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:46.507158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da56638b-777b-4fe2-ba35-bd753057a3d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:46.527102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-da56638b-777b-4fe2-ba35-bd753057a3d6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:46.528325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-da56638b-777b-4fe2-ba35-bd753057a3d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:46.609389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 31/59] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:46.633984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb49ca-9d93-4eed-8834-29a470ec228e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:46.689062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:46.766364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb49ca-9d93-4eed-8834-29a470ec228e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:46.803516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f5ca62-7050-4ce2-a3ec-cd4d52bdc2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:46.804156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f5ca62-7050-4ce2-a3ec-cd4d52bdc2ff None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:46.804441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f5ca62-7050-4ce2-a3ec-cd4d52bdc2ff None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:46.804721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f5ca62-7050-4ce2-a3ec-cd4d52bdc2ff None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:46.806013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f5ca62-7050-4ce2-a3ec-cd4d52bdc2ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:46.807543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f5ca62-7050-4ce2-a3ec-cd4d52bdc2ff None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:46.809165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 29/60] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:46.815264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da56638b-777b-4fe2-ba35-bd753057a3d6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:46.832672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:46.924035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da56638b-777b-4fe2-ba35-bd753057a3d6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:46.959834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afce7114-3604-434d-8e5c-bf669029a1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:46.960448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afce7114-3604-434d-8e5c-bf669029a1d7 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:46.960725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afce7114-3604-434d-8e5c-bf669029a1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:46.961000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afce7114-3604-434d-8e5c-bf669029a1d7 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:46.961693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afce7114-3604-434d-8e5c-bf669029a1d7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:46.963201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afce7114-3604-434d-8e5c-bf669029a1d7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:46.992690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 32/61] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:56:46 2022] POST /identity/v3/users => generated 275 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:56:50.174915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09e5d-7cae-4329-ae8e-b479e53642cf None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.175876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09e5d-7cae-4329-ae8e-b479e53642cf None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.176360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09e5d-7cae-4329-ae8e-b479e53642cf None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.177911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 30/62] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:50.185487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76720b43-1ae9-4285-973f-5fe1cf486318 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.186291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76720b43-1ae9-4285-973f-5fe1cf486318 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.186758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76720b43-1ae9-4285-973f-5fe1cf486318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.220025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-76720b43-1ae9-4285-973f-5fe1cf486318 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:50.221914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-76720b43-1ae9-4285-973f-5fe1cf486318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:50.258404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228404ac-f33b-4ad1-ac45-7947b46b6584 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.259051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228404ac-f33b-4ad1-ac45-7947b46b6584 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.259554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228404ac-f33b-4ad1-ac45-7947b46b6584 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.261121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 31/63] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:50.267301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2af10a-7ea0-42fe-b232-f53c4bc6bb04 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.267839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2af10a-7ea0-42fe-b232-f53c4bc6bb04 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.268317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2af10a-7ea0-42fe-b232-f53c4bc6bb04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.288999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-8e2af10a-7ea0-42fe-b232-f53c4bc6bb04 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:50.292136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8e2af10a-7ea0-42fe-b232-f53c4bc6bb04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:50.322016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:50.328220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 33/64] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:56:50.376429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76720b43-1ae9-4285-973f-5fe1cf486318 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:50.377424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 32/65] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:56:50.404744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2af10a-7ea0-42fe-b232-f53c4bc6bb04 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:50.471183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:50.532499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76720b43-1ae9-4285-973f-5fe1cf486318 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:50.557390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2af10a-7ea0-42fe-b232-f53c4bc6bb04 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:50.573424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d633c84-9920-4b85-bc74-cff20b88d2fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:50.574374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d633c84-9920-4b85-bc74-cff20b88d2fc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.574794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d633c84-9920-4b85-bc74-cff20b88d2fc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.575225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d633c84-9920-4b85-bc74-cff20b88d2fc None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.586277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d633c84-9920-4b85-bc74-cff20b88d2fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-7d633c84-9920-4b85-bc74-cff20b88d2fc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:50.591619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:50.592639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:50.593541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:50.596375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 34/66] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:56:50.598612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000ade5-e73a-44da-bc06-9b296ec828c6 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.598739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0545bf3-3dbd-4c52-b19b-d8836c5074cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:50.599186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000ade5-e73a-44da-bc06-9b296ec828c6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.599370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0545bf3-3dbd-4c52-b19b-d8836c5074cc None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.599652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0545bf3-3dbd-4c52-b19b-d8836c5074cc None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.599886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000ade5-e73a-44da-bc06-9b296ec828c6 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.599986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0545bf3-3dbd-4c52-b19b-d8836c5074cc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.601579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 35/67] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:50.606255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a000ade5-e73a-44da-bc06-9b296ec828c6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:50.615874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0545bf3-3dbd-4c52-b19b-d8836c5074cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-f0545bf3-3dbd-4c52-b19b-d8836c5074cc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:50.622710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:50.623806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:50.624772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:56:50.625894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 33/68] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:56:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:56:50.629682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdb4118-3c9f-47d4-a7ff-71166fb305ed None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.630093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdb4118-3c9f-47d4-a7ff-71166fb305ed None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.630764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdb4118-3c9f-47d4-a7ff-71166fb305ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.653261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-efdb4118-3c9f-47d4-a7ff-71166fb305ed None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:50.655563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-efdb4118-3c9f-47d4-a7ff-71166fb305ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:50.675884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:50.724949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a000ade5-e73a-44da-bc06-9b296ec828c6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:50.746996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 34/69] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:50.746996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdb4118-3c9f-47d4-a7ff-71166fb305ed None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:50.762756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ca5846-80f9-43d0-9daf-08435755145d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:50.763690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ca5846-80f9-43d0-9daf-08435755145d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.764032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ca5846-80f9-43d0-9daf-08435755145d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.764384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ca5846-80f9-43d0-9daf-08435755145d None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.765181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ca5846-80f9-43d0-9daf-08435755145d None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:50.765677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ca5846-80f9-43d0-9daf-08435755145d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:50.767540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ca5846-80f9-43d0-9daf-08435755145d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:50.776208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 36/70] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:56:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:50.780895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3611efc6-1621-45f0-8819-39c66fed275e None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.781415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:50.781670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3611efc6-1621-45f0-8819-39c66fed275e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.781959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3611efc6-1621-45f0-8819-39c66fed275e None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.783637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 37/71] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:50.787582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3611efc6-1621-45f0-8819-39c66fed275e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:50.860372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdb4118-3c9f-47d4-a7ff-71166fb305ed None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:50.892135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3611efc6-1621-45f0-8819-39c66fed275e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:50.895824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:50.896460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.896741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.897011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.897733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:50.898034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:50.899470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:50.908265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 35/72] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:50.909058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:50.915168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:50.927768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f19477-bb64-4a61-8595-29659fa5443b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:50.928480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f19477-bb64-4a61-8595-29659fa5443b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.928765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f19477-bb64-4a61-8595-29659fa5443b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.929041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f19477-bb64-4a61-8595-29659fa5443b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.939852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f19477-bb64-4a61-8595-29659fa5443b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-31f19477-bb64-4a61-8595-29659fa5443b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:50.945230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:50.946220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 38/73] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:56:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:50.947788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc9f261-1225-4aff-882a-97aabe3ba430 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:50.948503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc9f261-1225-4aff-882a-97aabe3ba430 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:50.948503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc9f261-1225-4aff-882a-97aabe3ba430 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:50.966612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ddc9f261-1225-4aff-882a-97aabe3ba430 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:50.967847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ddc9f261-1225-4aff-882a-97aabe3ba430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:51.001317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.011346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597da9e9-de92-4c20-b15c-b46cff80c652 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.036052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 39/74] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:51.037259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75523b3d-3db2-4c5b-bc03-7f618732f742 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.037547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75523b3d-3db2-4c5b-bc03-7f618732f742 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.037826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75523b3d-3db2-4c5b-bc03-7f618732f742 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.043152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 40/75] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:51.043152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75523b3d-3db2-4c5b-bc03-7f618732f742 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.053906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.054571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.054857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.055183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.071086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.072592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.074238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 36/76] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:50 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:51.075027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.145987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75523b3d-3db2-4c5b-bc03-7f618732f742 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.161217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.173125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2196c6-454d-4abf-977e-c9597886a33d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.187287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344df7d9-82b5-465e-86e1-3307337b8e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.188062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344df7d9-82b5-465e-86e1-3307337b8e69 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.188221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344df7d9-82b5-465e-86e1-3307337b8e69 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.188500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344df7d9-82b5-465e-86e1-3307337b8e69 None admin] PATH_INFO: `/v3/users/84df6af7bcfb4cfca1045798a2ef6d69` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.189731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344df7d9-82b5-465e-86e1-3307337b8e69 None admin] RBAC: Authorizing `identity:get_user(user_id=84df6af7bcfb4cfca1045798a2ef6d69)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.191260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344df7d9-82b5-465e-86e1-3307337b8e69 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.192902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 41/77] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/users/84df6af7bcfb4cfca1045798a2ef6d69 => generated 247 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:51.198633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca21d9-310e-44a2-b642-d95c6c9042fe None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.198929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca21d9-310e-44a2-b642-d95c6c9042fe None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.199235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca21d9-310e-44a2-b642-d95c6c9042fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.210397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.211062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.211342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.211619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.212340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:51.212643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.214021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.216934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-62ca21d9-310e-44a2-b642-d95c6c9042fe None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:51.218153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-62ca21d9-310e-44a2-b642-d95c6c9042fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:51.222476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 37/78] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:51.223322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.239274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.286933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 42/79] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:51.287740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca21d9-310e-44a2-b642-d95c6c9042fe None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.330206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1a279a-e173-4f47-86d7-8a0fb560ea3a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.363197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.363850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.364144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.364424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.365597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.367085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.368684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 38/80] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:51.369450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.403990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca21d9-310e-44a2-b642-d95c6c9042fe None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.414516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.437737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.438370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.438655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.438932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.448192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.473010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003dc70e-51ae-455c-9ec2-90edd0acffa9 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.508796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.509434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.509719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.509999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.512426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.513986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.580642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.582072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.611065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 43/81] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:56:51.611065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.622973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.664158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.686656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 39/82] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:56:51.687527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.716411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ec07da-3192-42d9-8669-2a8dcd2e7a85 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.760870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.761527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.761807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.762087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.763337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.764806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.766452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 44/83] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:51.767226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.798974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea444f51-8dfa-4494-83b6-c705ad828ebe None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.837590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.838155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.838439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.838717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.849198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.849357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:51.854254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:51.855184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:56:51.855926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 40/84] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:51.856556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.876813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bce6f7-1189-4ce5-b0d1-64aa46d93397 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:51.895615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:51.911515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:51.912142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:51.912422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:51.912702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:51.913388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:51.914842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:51.937478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 45/85] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:56:51 2022] POST /identity/v3/users => generated 289 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:56:51.938310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:51.973153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb9f33f-2802-427e-bc89-152e278d66c9 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.018187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.018865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.019185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.019461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.020733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.022276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.024086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 41/86] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:52.024978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.068496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e0bfaf-62be-4bd6-bbeb-d184c467ae94 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.114650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.115540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.115820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.116095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.117290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.118757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.120494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 46/87] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:51 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:52.121371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.124271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:52.146822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed6b240-2223-425a-806a-4f620de21d2f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.155854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:52.189633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.190246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.190534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.190816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.250494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e043cad-ec7c-4571-abbb-b50a24325dce None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.281974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.283528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.292893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.294386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.295012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.295473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.296700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:52.297473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.299420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.310455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 47/88] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:52.316544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.395722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:52.408121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 42/89] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:56:52.409244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.417960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:52.443817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f88110-1054-40f7-9ff8-b57b4b371437 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.480999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.481201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.481460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.481624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.482477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=f7f19a1e144b461c9f801f3d04a0e422) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:52.482796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.484300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.490479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 48/90] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/role_assignments?user.id=f7f19a1e144b461c9f801f3d04a0e422&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:52.499231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.563786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb89950-ecc0-4261-95bf-bc76ec454113 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.621476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752524e9-93bc-479c-80f2-b038c57d34f1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.636265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.636918 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.637303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.637587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.638456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=84df6af7bcfb4cfca1045798a2ef6d69) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:52.638772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.640402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.651590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:52.666679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 43/91] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/role_assignments?user.id=84df6af7bcfb4cfca1045798a2ef6d69&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:52.668102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.668708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.669001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.669285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.670325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.671855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.676297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.677975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 49/92] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:52.684784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.751697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:52.785046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25fc964-b4e4-4e66-b7ad-d9f3d02b4d93 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.801445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a51b3d-40ec-4aef-b22e-17636c7d1c98 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:52.839576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.840265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.840552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.840829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.843818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.845360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:52.847608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 44/93] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:52.854247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:52.870720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:52.871406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:52.871744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:52.872019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:52.955431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:52.962029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:52.963917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:53.109173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075d594f-1fb9-4260-85d4-4f819b1899d3 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:53.169180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 50/94] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 485 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:56:53.174886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:53.215231 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:53.215789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:53.216074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:53.216385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:53.217372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] RBAC: Adding query filter params (role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=f7f19a1e144b461c9f801f3d04a0e422) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:53.217693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:53.219192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:53.234140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 45/95] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:56:52 2022] GET /identity/v3/role_assignments?user.id=f7f19a1e144b461c9f801f3d04a0e422&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=629444c0e927466f8da9275a0841de95 => generated 259 bytes in 380 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:53.307476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:53.313408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219c520-8c3f-4b92-93ce-b427f1597d9c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:53.350470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23450f7b-ffed-42d6-967f-b7a1c2c46f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:53.351428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23450f7b-ffed-42d6-967f-b7a1c2c46f1b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:53.351848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23450f7b-ffed-42d6-967f-b7a1c2c46f1b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:53.352287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23450f7b-ffed-42d6-967f-b7a1c2c46f1b None admin] PATH_INFO: `/v3/projects/057795b728804191b0733d65e696fe85` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:53.353626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23450f7b-ffed-42d6-967f-b7a1c2c46f1b None admin] RBAC: Authorizing `identity:get_project(project_id=057795b728804191b0733d65e696fe85)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:53.355303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23450f7b-ffed-42d6-967f-b7a1c2c46f1b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:53.357265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 51/96] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:53 2022] GET /identity/v3/projects/057795b728804191b0733d65e696fe85 => generated 312 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:53.364417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:53.370980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:53.515311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d241b0-3b47-410b-8fd1-5eb2ccd25c17 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:53.552014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab678f6b-8ba5-4d6b-a246-d8c2a3135e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:53.553062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab678f6b-8ba5-4d6b-a246-d8c2a3135e04 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:53.553678 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab678f6b-8ba5-4d6b-a246-d8c2a3135e04 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:53.554149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab678f6b-8ba5-4d6b-a246-d8c2a3135e04 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:53.555249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab678f6b-8ba5-4d6b-a246-d8c2a3135e04 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=057795b728804191b0733d65e696fe85, user.id=f7f19a1e144b461c9f801f3d04a0e422) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:53.555762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab678f6b-8ba5-4d6b-a246-d8c2a3135e04 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:53.557463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab678f6b-8ba5-4d6b-a246-d8c2a3135e04 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:53.571162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 46/97] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:56:53 2022] GET /identity/v3/role_assignments?user.id=f7f19a1e144b461c9f801f3d04a0e422&scope.project.id=057795b728804191b0733d65e696fe85&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 259 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:54.340726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce4b3b0-5875-4397-b2d3-2798eb21c9fc None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:54.341794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce4b3b0-5875-4397-b2d3-2798eb21c9fc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:54.342272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce4b3b0-5875-4397-b2d3-2798eb21c9fc None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:54.343932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 52/98] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:56:54.349674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1d2b7-a106-4772-ae99-52e52a3eca1c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:54.350262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1d2b7-a106-4772-ae99-52e52a3eca1c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:54.350716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1d2b7-a106-4772-ae99-52e52a3eca1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:54.374213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d7c1d2b7-a106-4772-ae99-52e52a3eca1c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:54.376258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d7c1d2b7-a106-4772-ae99-52e52a3eca1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:54.451338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:54.458715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 47/99] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:54.504499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce4b3b0-5875-4397-b2d3-2798eb21c9fc None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:54.636767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce4b3b0-5875-4397-b2d3-2798eb21c9fc None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:54.672306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5b864a-793f-4f27-87a8-8eceb7992f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:54.672969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5b864a-793f-4f27-87a8-8eceb7992f44 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:54.673252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5b864a-793f-4f27-87a8-8eceb7992f44 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:54.673527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5b864a-793f-4f27-87a8-8eceb7992f44 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:54.685206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5b864a-793f-4f27-87a8-8eceb7992f44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-7f5b864a-793f-4f27-87a8-8eceb7992f44 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:54.690164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:54.691046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:54.691804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 53/100] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:54.700165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c1d2b7-a106-4772-ae99-52e52a3eca1c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:54.806663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a3662-8ec6-412d-84d9-9590b4aa3639 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:54.807320 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a3662-8ec6-412d-84d9-9590b4aa3639 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:54.807787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a3662-8ec6-412d-84d9-9590b4aa3639 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:54.809090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c1d2b7-a106-4772-ae99-52e52a3eca1c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:54.809914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 54/101] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:54.815373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33f3bb-eebb-436b-8d95-8637d7a33fea None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:54.815975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33f3bb-eebb-436b-8d95-8637d7a33fea None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:54.816445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33f3bb-eebb-436b-8d95-8637d7a33fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:54.839377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ff33f3bb-eebb-436b-8d95-8637d7a33fea None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:54.841858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ff33f3bb-eebb-436b-8d95-8637d7a33fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:54.869601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:54.880206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:54.880858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:54.881142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:54.881423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:54.882161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:54.882468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:54.883889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:54.906678 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 48/102] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:56:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:54.920581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:54.921670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 55/103] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:56:54.944405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff33f3bb-eebb-436b-8d95-8637d7a33fea None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:55.048391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff33f3bb-eebb-436b-8d95-8637d7a33fea None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:55.085214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765e9585-b380-487a-826b-374175a19e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:55.086272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765e9585-b380-487a-826b-374175a19e99 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:55.086743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765e9585-b380-487a-826b-374175a19e99 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:55.087230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765e9585-b380-487a-826b-374175a19e99 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:55.088593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765e9585-b380-487a-826b-374175a19e99 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:55.090268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765e9585-b380-487a-826b-374175a19e99 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:55.092840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 56/104] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:55.099529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765e9585-b380-487a-826b-374175a19e99 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:55.101151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:55.131366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:55.208441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abe4898-f282-4520-8e03-9b0ec328ddd4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:55.211242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765e9585-b380-487a-826b-374175a19e99 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:55.246429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:55.247508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:55.247987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:55.248451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:55.249359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:55.251034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:55.256390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:55.257112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:55.257397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:55.257936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] PATH_INFO: `/v3/users/ea72826f95f944ff880b7e8905da85d4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:55.280450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] RBAC: Authorizing `identity:get_user(user_id=ea72826f95f944ff880b7e8905da85d4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:55.281963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:55.283858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 49/105] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:54 2022] GET /identity/v3/users/ea72826f95f944ff880b7e8905da85d4 => generated 289 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:55.287271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 57/106] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:56:55 2022] POST /identity/v3/projects => generated 302 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:56:55.291158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:55.446972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3c9051-92a4-485a-afad-5e6a167d0904 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:55.469384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:55.506669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39484832-3207-4f77-aea6-5ed74e119b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:55.507378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39484832-3207-4f77-aea6-5ed74e119b1d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:55.507666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39484832-3207-4f77-aea6-5ed74e119b1d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:55.507947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39484832-3207-4f77-aea6-5ed74e119b1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:55.593887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39484832-3207-4f77-aea6-5ed74e119b1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:55.595446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39484832-3207-4f77-aea6-5ed74e119b1d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:55.722278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 50/107] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:56:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:56:55.728221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:55.773502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90485909-8502-4ff0-b93e-629efb887938 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:55.773873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90485909-8502-4ff0-b93e-629efb887938 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:55.774036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90485909-8502-4ff0-b93e-629efb887938 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:55.775445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 51/108] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:56:55.780843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3d2ec-d92d-454d-9ddd-d2a157d6d2d3 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:55.781099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3d2ec-d92d-454d-9ddd-d2a157d6d2d3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:55.781328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3d2ec-d92d-454d-9ddd-d2a157d6d2d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:55.816017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-05d3d2ec-d92d-454d-9ddd-d2a157d6d2d3 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:55.817242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-05d3d2ec-d92d-454d-9ddd-d2a157d6d2d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:55.826668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:55.835920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:55.837808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2917bff-72c6-4f25-91c6-68310ac3f095 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:55.880781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:55.881830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:55.883099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:55.883574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:55.890656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:55.892440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:55.894359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 58/109] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:55 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:55.908056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:55.977768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 52/110] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:56:56.012276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3d2ec-d92d-454d-9ddd-d2a157d6d2d3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:56.099201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75cf8ca-d5e6-4658-86ed-3bb637d3e910 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:56.166655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3d2ec-d92d-454d-9ddd-d2a157d6d2d3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:56.183750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881323ef-d84a-43f3-83d9-5e060f8a38cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:56.184413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881323ef-d84a-43f3-83d9-5e060f8a38cd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.184701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881323ef-d84a-43f3-83d9-5e060f8a38cd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.185023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881323ef-d84a-43f3-83d9-5e060f8a38cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.185850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881323ef-d84a-43f3-83d9-5e060f8a38cd None admin] RBAC: Adding query filter params (role.id=7b22f4d466c74886bb0103bf3f7b59d1, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=ea72826f95f944ff880b7e8905da85d4) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:56.186169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881323ef-d84a-43f3-83d9-5e060f8a38cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:56.187674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881323ef-d84a-43f3-83d9-5e060f8a38cd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:56.196651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 59/111] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:56:55 2022] GET /identity/v3/role_assignments?user.id=ea72826f95f944ff880b7e8905da85d4&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=7b22f4d466c74886bb0103bf3f7b59d1 => generated 259 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:56.197921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381bab1-b9f7-4b89-996e-17ad67ea8def None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.198208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381bab1-b9f7-4b89-996e-17ad67ea8def None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.198488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d381bab1-b9f7-4b89-996e-17ad67ea8def None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.199755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 60/112] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:56.205440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b06733-955b-4df3-9695-d0b5691f1c7c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.205736 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b06733-955b-4df3-9695-d0b5691f1c7c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.206017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b06733-955b-4df3-9695-d0b5691f1c7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.206588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:56.207511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.207964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.208403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.221375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:56.228367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:56.229239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:56.229987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:56:56.232488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 53/113] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:56.241601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:56.242794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:56.243889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-92b06733-955b-4df3-9695-d0b5691f1c7c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:56.245283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-92b06733-955b-4df3-9695-d0b5691f1c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:56.264835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:56.360474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 61/114] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 26 02:56:56.385262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcd7928-cd3a-48be-b5bc-70f7ebd25e50 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:56.408056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b06733-955b-4df3-9695-d0b5691f1c7c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:56.427875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:56.428525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.428807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.429084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.429850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:56.430174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:56.432344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:56.442394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 54/115] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:56:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:56.448771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:56.582030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:56.623550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9e5541-ecf2-4f8b-8a3b-749647de701c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:56.636406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b06733-955b-4df3-9695-d0b5691f1c7c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:56.672294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa03ef99-01fa-4eec-961d-de74eb5c4082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:56.673437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa03ef99-01fa-4eec-961d-de74eb5c4082 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.673862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa03ef99-01fa-4eec-961d-de74eb5c4082 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.674367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa03ef99-01fa-4eec-961d-de74eb5c4082 None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.675961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa03ef99-01fa-4eec-961d-de74eb5c4082 None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:56.677206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756bce06-3382-40d0-854b-004e0aeecb74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:56.677706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa03ef99-01fa-4eec-961d-de74eb5c4082 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:56.678310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756bce06-3382-40d0-854b-004e0aeecb74 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.678784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756bce06-3382-40d0-854b-004e0aeecb74 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.679295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756bce06-3382-40d0-854b-004e0aeecb74 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.683026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 55/116] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:56 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:56.684740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fef47f-9b16-452c-8b6b-408e229051f9 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.685054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fef47f-9b16-452c-8b6b-408e229051f9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.685339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fef47f-9b16-452c-8b6b-408e229051f9 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.686741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 56/117] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:56.687677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9030e1-2448-4297-bd6e-77ad36e32359 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.687969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9030e1-2448-4297-bd6e-77ad36e32359 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.688243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9030e1-2448-4297-bd6e-77ad36e32359 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.689552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 57/118] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:56.690216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:56.691136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9030e1-2448-4297-bd6e-77ad36e32359 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:56.704968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756bce06-3382-40d0-854b-004e0aeecb74 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-756bce06-3382-40d0-854b-004e0aeecb74 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:56.711849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:56.712693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:56:56.713444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:56.715886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 62/119] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:56:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:56.717396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d78d8-5548-492b-95bf-188ceb8f773e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.717878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d78d8-5548-492b-95bf-188ceb8f773e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.718387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21d78d8-5548-492b-95bf-188ceb8f773e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.738237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b21d78d8-5548-492b-95bf-188ceb8f773e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:56.740212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b21d78d8-5548-492b-95bf-188ceb8f773e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:56.808419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9030e1-2448-4297-bd6e-77ad36e32359 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:56.815562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 63/120] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:56.817671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ebbe64-cab8-4232-8e11-e12d472bf507 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.818379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ebbe64-cab8-4232-8e11-e12d472bf507 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.819088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ebbe64-cab8-4232-8e11-e12d472bf507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.845980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d2ebbe64-cab8-4232-8e11-e12d472bf507 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:56.848526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d2ebbe64-cab8-4232-8e11-e12d472bf507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:56.870766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:56.871479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:56.871804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:56.872088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:56.873344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:56.875031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:56.883119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 58/121] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:56 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:56.884938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:56.890237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:56.926030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 64/122] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:56.927319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ebbe64-cab8-4232-8e11-e12d472bf507 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:56.987321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:57.032504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94def328-1ab1-4c18-9ce9-f1224b1513f7 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.047045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ebbe64-cab8-4232-8e11-e12d472bf507 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.072855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.073493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.073840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.074096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.074925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:57.075278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.077024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.084605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.085565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.086060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.086525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.091301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 59/123] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:57.092700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.104892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:57.109943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:57.110824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:57.111632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 65/124] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:57.112243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.180360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:57.215708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9f84a-857c-4a6b-aee3-858ace12e3b2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.222863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31f6d29-614c-40fa-b21c-ca3e005b6a35 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.235737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:57.254354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.255078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.255380 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.255686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.264802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.265842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.266415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.266879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/f7f19a1e144b461c9f801f3d04a0e422/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.267375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:57.269450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] RBAC: Authorizing `identity:create_grant(project_id=1e52b22e640c409e9307765a7cc712c4, user_id=f7f19a1e144b461c9f801f3d04a0e422, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.271186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:57.272879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:57.274267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 66/125] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:57.275063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.295241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 60/126] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:56:57 2022] PUT /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/f7f19a1e144b461c9f801f3d04a0e422/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:56:57.301320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.389124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6e4b63-fd20-4e75-bc11-2322ba762d82 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.431142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.433020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.433020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.433311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] PATH_INFO: `/v3/users/84df6af7bcfb4cfca1045798a2ef6d69` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.435830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] RBAC: Authorizing `identity:get_user(user_id=84df6af7bcfb4cfca1045798a2ef6d69)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.437618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.439586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 67/127] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/users/84df6af7bcfb4cfca1045798a2ef6d69 => generated 247 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:57.440233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.506526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faf8e74-ae72-44ad-98e8-c41c3765ac7d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.553493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:57.573321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:57.587309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.587965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.588249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.588530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.589337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:57.589648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.591139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.600676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 61/128] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:57.601536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.646034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed707f5-a826-4322-a343-0fab3351f9f4 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.714019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602842f9-4bd6-42fb-a0e9-08a9e9386b0e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.727682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.728336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.728620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.728900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.729635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:57.729947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.731432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.746856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 68/129] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:57.748393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.752653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.753674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.754178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.754643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.756078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.757739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.759756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 62/130] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:57.760721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.828517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:57.866188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310f3e82-5f78-419e-b927-d803087edf01 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.902034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.903117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.903608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.904507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.906047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.907976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.909819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 63/131] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:57.911047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:57.912316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66337206-6d36-4d6a-af3a-3afea101dc51 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:57.924005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:57.976795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:57.977446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:57.977727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:57.977999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:57.979213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:57.980702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:57.982501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 69/132] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:57.986971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:58.042011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ca97f-a0bb-4681-aad0-443f62569e4a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:58.066792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:58.078639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381b0c7b-1e1f-40ed-b309-ce75b59cbab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:58.079549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381b0c7b-1e1f-40ed-b309-ce75b59cbab1 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.079836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381b0c7b-1e1f-40ed-b309-ce75b59cbab1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.080120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381b0c7b-1e1f-40ed-b309-ce75b59cbab1 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/84df6af7bcfb4cfca1045798a2ef6d69/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.082091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381b0c7b-1e1f-40ed-b309-ce75b59cbab1 None admin] RBAC: Authorizing `identity:create_grant(project_id=1e52b22e640c409e9307765a7cc712c4, user_id=84df6af7bcfb4cfca1045798a2ef6d69, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:58.083593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381b0c7b-1e1f-40ed-b309-ce75b59cbab1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:58.100484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 64/133] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:56:57 2022] PUT /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/84df6af7bcfb4cfca1045798a2ef6d69/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:56:58.102032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e3ef3a-c503-4fef-9d89-dbe67087dd54 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.102322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e3ef3a-c503-4fef-9d89-dbe67087dd54 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.102602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e3ef3a-c503-4fef-9d89-dbe67087dd54 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.110998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 65/134] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:56:58.110998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e3ef3a-c503-4fef-9d89-dbe67087dd54 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:58.134263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7aec0e-0fb8-443b-9846-027dabef2ef0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:58.177444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94db36a-0eb5-47c4-8bbd-9965438f3c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:58.178371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94db36a-0eb5-47c4-8bbd-9965438f3c27 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.178801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94db36a-0eb5-47c4-8bbd-9965438f3c27 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.179424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94db36a-0eb5-47c4-8bbd-9965438f3c27 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.180929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94db36a-0eb5-47c4-8bbd-9965438f3c27 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:58.182597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94db36a-0eb5-47c4-8bbd-9965438f3c27 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:58.184703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 70/135] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:57 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:58.186172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc0e557-68e2-4e14-8232-4bfed90c7cd8 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.186632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc0e557-68e2-4e14-8232-4bfed90c7cd8 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.187068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc0e557-68e2-4e14-8232-4bfed90c7cd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.208712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9bc0e557-68e2-4e14-8232-4bfed90c7cd8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:58.210400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9bc0e557-68e2-4e14-8232-4bfed90c7cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:58.251629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:58.286087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 71/136] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:58.287303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc0e557-68e2-4e14-8232-4bfed90c7cd8 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:58.302808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e3ef3a-c503-4fef-9d89-dbe67087dd54 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:58.338612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:58.339322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.339606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.339885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] PATH_INFO: `/v3/projects/057795b728804191b0733d65e696fe85` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.341108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] RBAC: Authorizing `identity:get_project(project_id=057795b728804191b0733d65e696fe85)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:58.342610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:58.344654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 66/137] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:58 2022] GET /identity/v3/projects/057795b728804191b0733d65e696fe85 => generated 312 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:58.345797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:58.402304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc0e557-68e2-4e14-8232-4bfed90c7cd8 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:58.439361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:58.440118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.440401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.440682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/f7f19a1e144b461c9f801f3d04a0e422/roles/629444c0e927466f8da9275a0841de95` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.442890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] RBAC: Authorizing `identity:create_grant(project_id=1e52b22e640c409e9307765a7cc712c4, user_id=f7f19a1e144b461c9f801f3d04a0e422, role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:58.444616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:58.460105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 72/138] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:56:58 2022] PUT /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/f7f19a1e144b461c9f801f3d04a0e422/roles/629444c0e927466f8da9275a0841de95 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:56:58.460912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:58.515596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:58.535513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:58.631871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555baad8-a84e-4606-9172-9ec75f70186b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:58.668295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:58.668987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.669301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.669555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] PATH_INFO: `/v3/projects/057795b728804191b0733d65e696fe85/users/f7f19a1e144b461c9f801f3d04a0e422/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.672575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] RBAC: Authorizing `identity:create_grant(project_id=057795b728804191b0733d65e696fe85, user_id=f7f19a1e144b461c9f801f3d04a0e422, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:58.674054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:58.695373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 73/139] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:56:58 2022] PUT /identity/v3/projects/057795b728804191b0733d65e696fe85/users/f7f19a1e144b461c9f801f3d04a0e422/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:56:58.706491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d13be3-26e2-441a-94aa-17169d61ccd2 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:58.756320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616e141e-b637-4205-a1d8-cec76ffd2dee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:58.756986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e141e-b637-4205-a1d8-cec76ffd2dee None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.757272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e141e-b637-4205-a1d8-cec76ffd2dee None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.757545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e141e-b637-4205-a1d8-cec76ffd2dee None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.759634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616e141e-b637-4205-a1d8-cec76ffd2dee None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:58.761175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616e141e-b637-4205-a1d8-cec76ffd2dee None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:58.763262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 67/140] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:56:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 418 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:58.769289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:58.848454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9fcb3-0437-4411-96ba-ce9472d17d1d None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.848803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9fcb3-0437-4411-96ba-ce9472d17d1d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.849051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9fcb3-0437-4411-96ba-ce9472d17d1d None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.850423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 68/141] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:56:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:56:58.859248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7570c40e-747a-4dee-a578-9e895adefd0b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.859601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7570c40e-747a-4dee-a578-9e895adefd0b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.859882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7570c40e-747a-4dee-a578-9e895adefd0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.883405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-7570c40e-747a-4dee-a578-9e895adefd0b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:56:58.884827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7570c40e-747a-4dee-a578-9e895adefd0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:56:58.896057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:58.915068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e62faa-72a1-4a0c-8dae-4ae21e1a0b6e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:58.989117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:58.992395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:58.993480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:58.994208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:58.996599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:59.000203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:59.014451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 69/142] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:56:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:56:59.040185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 74/143] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:56:58 2022] POST /identity/v3/users => generated 283 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:56:59.048680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7570c40e-747a-4dee-a578-9e895adefd0b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:59.128352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:59.197945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7570c40e-747a-4dee-a578-9e895adefd0b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:59.241027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:59.242024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:59.242499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:59.243012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:59.255544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:56:59.261082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:56:59.261980 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:56:59.262742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:56:59.265357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 70/144] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:56:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:56:59.271470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:59.387545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e945dad-1c79-4b3e-9c2f-b2414517e750 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:59.431541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:59.432424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:59.432842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:59.433242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:59.434374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:56:59.434842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:59.437004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:59.449459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 75/145] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:56:59 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:56:59.456852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:59.567478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1b6380-6e3e-4a05-a9f4-ad592696f3d1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:59.579251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:59.614000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:59.615061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:59.615540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:59.616003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] PATH_INFO: `/v3/users/ea72826f95f944ff880b7e8905da85d4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:59.623789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] RBAC: Authorizing `identity:get_user(user_id=ea72826f95f944ff880b7e8905da85d4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:59.625572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:59.627555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 71/146] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:56:59 2022] GET /identity/v3/users/ea72826f95f944ff880b7e8905da85d4 => generated 289 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:59.633281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:59.656758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:56:59.778589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca87091b-caf1-4b81-999b-88879b95c309 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:59.815627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ac4ef7-9e7e-4575-8435-f42b4e0c1e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:59.816285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac4ef7-9e7e-4575-8435-f42b4e0c1e10 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:59.816604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac4ef7-9e7e-4575-8435-f42b4e0c1e10 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:59.816878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ac4ef7-9e7e-4575-8435-f42b4e0c1e10 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:59.818170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ac4ef7-9e7e-4575-8435-f42b4e0c1e10 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:59.819640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ac4ef7-9e7e-4575-8435-f42b4e0c1e10 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:56:59.821664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 76/147] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:56:59 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:56:59.827482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:56:59.942840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1b5648-29b2-4d6d-81e6-7d0efc9d68b4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:56:59.977934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957343d7-3570-4452-a204-11ea1950a26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:56:59.978615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957343d7-3570-4452-a204-11ea1950a26c None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:56:59.978893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957343d7-3570-4452-a204-11ea1950a26c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:56:59.979206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957343d7-3570-4452-a204-11ea1950a26c None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/ea72826f95f944ff880b7e8905da85d4/roles/7b22f4d466c74886bb0103bf3f7b59d1` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:56:59.981144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957343d7-3570-4452-a204-11ea1950a26c None admin] RBAC: Authorizing `identity:create_grant(project_id=1e52b22e640c409e9307765a7cc712c4, user_id=ea72826f95f944ff880b7e8905da85d4, role_id=7b22f4d466c74886bb0103bf3f7b59d1)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:56:59.982671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957343d7-3570-4452-a204-11ea1950a26c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:00.000553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 72/148] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:56:59 2022] PUT /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4/users/ea72826f95f944ff880b7e8905da85d4/roles/7b22f4d466c74886bb0103bf3f7b59d1 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:00.342369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd166f6-cad1-400b-98db-9bde0091bf42 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:00.342775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd166f6-cad1-400b-98db-9bde0091bf42 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:00.342983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd166f6-cad1-400b-98db-9bde0091bf42 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:00.344524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 77/149] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:00.350119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdb4907-d592-49ef-a709-1bf2ff1fb412 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:00.350408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdb4907-d592-49ef-a709-1bf2ff1fb412 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:00.350690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdb4907-d592-49ef-a709-1bf2ff1fb412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:00.375359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-dcdb4907-d592-49ef-a709-1bf2ff1fb412 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:00.376563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dcdb4907-d592-49ef-a709-1bf2ff1fb412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:00.414521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:00.456732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 73/150] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:00.487903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd166f6-cad1-400b-98db-9bde0091bf42 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:00.592887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:00.624987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd166f6-cad1-400b-98db-9bde0091bf42 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:00.666992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:00.667683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:00.667982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:00.668321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:00.680961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:00.686764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:00.687674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:00.688455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 78/151] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:00.700127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdb4907-d592-49ef-a709-1bf2ff1fb412 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:00.841836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdb4907-d592-49ef-a709-1bf2ff1fb412 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:00.885336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:00.885973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:00.886250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:00.886525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:00.887276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:00.887578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:00.889066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:00.899999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 74/152] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:00.907827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:01.055357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d91b1cf-25b5-421c-98a0-2ffeec16f17c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:01.101001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c5d4f0-20f1-4170-9a7d-0caba902445a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:01.101671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c5d4f0-20f1-4170-9a7d-0caba902445a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:01.101954 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c5d4f0-20f1-4170-9a7d-0caba902445a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:01.102236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c5d4f0-20f1-4170-9a7d-0caba902445a None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:01.103466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c5d4f0-20f1-4170-9a7d-0caba902445a None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:01.104957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c5d4f0-20f1-4170-9a7d-0caba902445a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:01.106591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 79/153] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:00 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:01.112408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:01.175210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:01.265860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3280f498-1146-4b5e-8531-0a6bb141f01b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:01.308764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:01.309401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:01.309678 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:01.310023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:01.385764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ac5ea-b06e-47fe-8609-0cc644203cd2 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:01.386102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ac5ea-b06e-47fe-8609-0cc644203cd2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:01.386333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ac5ea-b06e-47fe-8609-0cc644203cd2 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:01.387744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 80/154] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:01.393605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a14dd4-ea00-44a1-85ea-8f43a8a785a2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:01.393887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a14dd4-ea00-44a1-85ea-8f43a8a785a2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:01.394163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a14dd4-ea00-44a1-85ea-8f43a8a785a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:01.415345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:01.423471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:01.425027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:01.431224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-49a14dd4-ea00-44a1-85ea-8f43a8a785a2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:01.439743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-49a14dd4-ea00-44a1-85ea-8f43a8a785a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:01.489986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:01.576479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 81/155] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:01.586705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 75/156] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:01.592598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a14dd4-ea00-44a1-85ea-8f43a8a785a2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:01.619852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:01.747257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a14dd4-ea00-44a1-85ea-8f43a8a785a2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:01.772775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440a8aae-3d0f-4fe3-aa9f-69575fd777e8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:01.800987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:01.801649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:01.801925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:01.802195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:01.807673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:01.809154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:01.811087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 82/157] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:01 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:01.816894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:01.819867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:01.822268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:01.832210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2128f95-7fa0-4552-89d6-cd3be9e1275b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:01.832893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2128f95-7fa0-4552-89d6-cd3be9e1275b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:01.833182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2128f95-7fa0-4552-89d6-cd3be9e1275b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:01.833465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2128f95-7fa0-4552-89d6-cd3be9e1275b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:01.852329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2128f95-7fa0-4552-89d6-cd3be9e1275b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a2128f95-7fa0-4552-89d6-cd3be9e1275b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:01.865722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:01.866728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:01.867559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:01.870730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 76/158] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:01.872211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8c112e-b9a9-4809-a559-c163829b2696 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:01.872561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8c112e-b9a9-4809-a559-c163829b2696 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:01.872830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8c112e-b9a9-4809-a559-c163829b2696 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:01.874457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 77/159] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:01.876273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8c112e-b9a9-4809-a559-c163829b2696 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.014509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8c112e-b9a9-4809-a559-c163829b2696 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:02.036760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185cee5e-38ed-485f-b0b6-8dbe4327f2de None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:02.055221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09b9f28-4cfd-4ccd-89e3-2bc5b610cddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:02.056231 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b9f28-4cfd-4ccd-89e3-2bc5b610cddf None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.056657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b9f28-4cfd-4ccd-89e3-2bc5b610cddf None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.057074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b9f28-4cfd-4ccd-89e3-2bc5b610cddf None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.058127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b9f28-4cfd-4ccd-89e3-2bc5b610cddf None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:02.058605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b9f28-4cfd-4ccd-89e3-2bc5b610cddf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:02.060256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b9f28-4cfd-4ccd-89e3-2bc5b610cddf None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:02.070165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 78/160] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:02.071845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6a0402-bba6-4247-b141-e8f5ac03f8b6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.072270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6a0402-bba6-4247-b141-e8f5ac03f8b6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.072681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6a0402-bba6-4247-b141-e8f5ac03f8b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.098408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1b6a0402-bba6-4247-b141-e8f5ac03f8b6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:02.100111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1b6a0402-bba6-4247-b141-e8f5ac03f8b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:02.108560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:02.109632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.109779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.110153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.111090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=f7f19a1e144b461c9f801f3d04a0e422) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:02.111519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:02.113166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:02.123838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 83/161] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:01 2022] GET /identity/v3/role_assignments?user.id=f7f19a1e144b461c9f801f3d04a0e422&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 608 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:02.125025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.187360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:02.195407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 79/162] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:02.242493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:02.244774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6a0402-bba6-4247-b141-e8f5ac03f8b6 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.291073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc85e905-2451-4fcb-bcbe-36deccfd6ff1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:02.355756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:02.357083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.357083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.357083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] PATH_INFO: `/v3/users/84df6af7bcfb4cfca1045798a2ef6d69` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.358306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] RBAC: Authorizing `identity:get_user(user_id=84df6af7bcfb4cfca1045798a2ef6d69)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:02.361399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:02.361554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 84/163] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:02 2022] GET /identity/v3/users/84df6af7bcfb4cfca1045798a2ef6d69 => generated 247 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:02.384726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.405347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6a0402-bba6-4247-b141-e8f5ac03f8b6 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:02.448791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:02.449888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.450519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.451012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.465041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:02.471207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:02.472067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:02.472826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:02.475335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 80/164] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:02.481594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.548188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:02.548571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916def4-7124-40e1-b3c6-78c4e62cbc31 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:02.603808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:02.606990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.606990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.606990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.609333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:02.634570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13022d13-635a-4df4-ba20-88d8bd672851 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:02.681947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:02.682979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.683474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.684100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.685120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:02.685627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:02.687336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:02.701050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 81/165] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:02.704528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:02.706082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:02.707254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.851717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 85/166] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:02.867018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.872020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c484d6-385d-48d3-8d8c-54c7bb06275d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:02.875584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:02.923254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:02.924265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:02.924763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:02.925227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:02.926676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:02.928460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:02.930610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 82/167] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:02 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:02.936040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:02.943323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:03.080363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e58fc56-0315-45e6-8c0c-798e5d7f92ee None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:03.118807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0d1a86-8cfa-4abb-9706-65e7f7d43daf None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:03.126492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:03.127518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.128015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.128496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.191780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:03.192471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.192756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.193022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.199411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:03.199916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:03.201575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:03.203614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 86/168] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:02 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:03.211973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:03.280035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:03.281522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:03.337666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e664b-b3e2-4b9f-8b23-2a8296871c98 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:03.340098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:03.390236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db5cf59-1c2f-489f-8a1b-3725dda69e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:03.391183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5cf59-1c2f-489f-8a1b-3725dda69e68 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.391616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5cf59-1c2f-489f-8a1b-3725dda69e68 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.392026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5cf59-1c2f-489f-8a1b-3725dda69e68 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.392987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5cf59-1c2f-489f-8a1b-3725dda69e68 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=84df6af7bcfb4cfca1045798a2ef6d69) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:03.393440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5cf59-1c2f-489f-8a1b-3725dda69e68 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:03.395239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5cf59-1c2f-489f-8a1b-3725dda69e68 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:03.408518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 87/169] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:03 2022] GET /identity/v3/role_assignments?user.id=84df6af7bcfb4cfca1045798a2ef6d69&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:03.410905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5fac33-fb2b-4800-bc8a-9ea9a21a8807 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.411537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5fac33-fb2b-4800-bc8a-9ea9a21a8807 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.413335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5fac33-fb2b-4800-bc8a-9ea9a21a8807 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.415851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 88/170] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:03.424504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272b228-f6cf-46cd-aa70-d8b06c0e3958 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.426383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272b228-f6cf-46cd-aa70-d8b06c0e3958 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.426826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272b228-f6cf-46cd-aa70-d8b06c0e3958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.469877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7272b228-f6cf-46cd-aa70-d8b06c0e3958 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:03.471565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7272b228-f6cf-46cd-aa70-d8b06c0e3958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:03.548643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 83/171] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 613 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:57:03.556281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:03.626559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 89/172] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:03.657288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272b228-f6cf-46cd-aa70-d8b06c0e3958 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:03.678312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:03.724264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e225de64-6ffb-43f0-b251-390de66a6253 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:03.814645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:03.815587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.816009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.816490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.817847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:03.819695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:03.823534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 84/173] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:03 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:03.844388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:03.856305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:03.864987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272b228-f6cf-46cd-aa70-d8b06c0e3958 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:03.914529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3730022a-fc78-411a-bff0-6d1bb06fbd0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:03.915502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3730022a-fc78-411a-bff0-6d1bb06fbd0f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.915925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3730022a-fc78-411a-bff0-6d1bb06fbd0f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.916356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3730022a-fc78-411a-bff0-6d1bb06fbd0f None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.930737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3730022a-fc78-411a-bff0-6d1bb06fbd0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-3730022a-fc78-411a-bff0-6d1bb06fbd0f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:03.937802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:03.938873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:03.939851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:03.941875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 90/174] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:03.944213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074021ca-60a1-4cad-b682-582e677dae79 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.944669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074021ca-60a1-4cad-b682-582e677dae79 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.945083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074021ca-60a1-4cad-b682-582e677dae79 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.946507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 91/175] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:03.947913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65479320-2ffb-4a82-8a80-f842792aa991 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:03.948339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65479320-2ffb-4a82-8a80-f842792aa991 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:03.948777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65479320-2ffb-4a82-8a80-f842792aa991 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:03.952845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 92/176] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:03.954519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65479320-2ffb-4a82-8a80-f842792aa991 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:03.987794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefa0917-b244-40b0-a2e8-2df5405e1c6c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.031523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd7d3a5-356d-4de4-a8a2-6f95c183cc86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.032583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd7d3a5-356d-4de4-a8a2-6f95c183cc86 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.033291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd7d3a5-356d-4de4-a8a2-6f95c183cc86 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.033720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd7d3a5-356d-4de4-a8a2-6f95c183cc86 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.034678 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd7d3a5-356d-4de4-a8a2-6f95c183cc86 None admin] RBAC: Adding query filter params (role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=f7f19a1e144b461c9f801f3d04a0e422) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:04.035169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd7d3a5-356d-4de4-a8a2-6f95c183cc86 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:04.037237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd7d3a5-356d-4de4-a8a2-6f95c183cc86 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:04.047510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 85/177] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:03 2022] GET /identity/v3/role_assignments?user.id=f7f19a1e144b461c9f801f3d04a0e422&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=629444c0e927466f8da9275a0841de95 => generated 608 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:04.054181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb7864-9765-4391-802d-f29ae53cd729 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.054181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb7864-9765-4391-802d-f29ae53cd729 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.054181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb7864-9765-4391-802d-f29ae53cd729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.070206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.092995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-dafb7864-9765-4391-802d-f29ae53cd729 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:04.094228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dafb7864-9765-4391-802d-f29ae53cd729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:04.136735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65479320-2ffb-4a82-8a80-f842792aa991 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.176538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 86/178] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:04.182609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43dba7c-dbb5-42a1-8f25-9b51e8d7aea3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.183293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43dba7c-dbb5-42a1-8f25-9b51e8d7aea3 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.183575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43dba7c-dbb5-42a1-8f25-9b51e8d7aea3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.183851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43dba7c-dbb5-42a1-8f25-9b51e8d7aea3 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.184662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43dba7c-dbb5-42a1-8f25-9b51e8d7aea3 None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:04.185051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43dba7c-dbb5-42a1-8f25-9b51e8d7aea3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:04.185153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d626ee7f-bc13-42a3-a0cf-d7d9b313da70 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.185529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d626ee7f-bc13-42a3-a0cf-d7d9b313da70 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.185912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d626ee7f-bc13-42a3-a0cf-d7d9b313da70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.186466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43dba7c-dbb5-42a1-8f25-9b51e8d7aea3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:04.196443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 93/179] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:04.197758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdcf5dc-263b-4c3e-9369-556ed7fc63fb None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.203509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdcf5dc-263b-4c3e-9369-556ed7fc63fb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.203874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdcf5dc-263b-4c3e-9369-556ed7fc63fb None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.206623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 94/180] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:04 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:04.207541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdcf5dc-263b-4c3e-9369-556ed7fc63fb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.225553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d626ee7f-bc13-42a3-a0cf-d7d9b313da70 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:04.227115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d626ee7f-bc13-42a3-a0cf-d7d9b313da70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:04.236451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.310659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 87/181] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:04.312643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e377f1-371c-4385-b754-3e91673e6005 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.312643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e377f1-371c-4385-b754-3e91673e6005 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.312643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e377f1-371c-4385-b754-3e91673e6005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.331076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-98e377f1-371c-4385-b754-3e91673e6005 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:04.332270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-98e377f1-371c-4385-b754-3e91673e6005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:04.332526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdcf5dc-263b-4c3e-9369-556ed7fc63fb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.385863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.386773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.387188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.387467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] PATH_INFO: `/v3/users/f7f19a1e144b461c9f801f3d04a0e422` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.388803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] RBAC: Authorizing `identity:get_user(user_id=f7f19a1e144b461c9f801f3d04a0e422)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:04.390280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:04.393012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 95/182] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/users/f7f19a1e144b461c9f801f3d04a0e422 => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:04.393802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.398700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.411253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 88/183] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:04.412134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e377f1-371c-4385-b754-3e91673e6005 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.493260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.506342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedafd06-ce50-4082-b4f1-267cb5a4f789 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.512970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e377f1-371c-4385-b754-3e91673e6005 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.542557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.543236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.543599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.543783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.550275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.550928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.551240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.551517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.553853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:04.559955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:04.560866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:04.561650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 96/184] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:04.562290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.563708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:04.568769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:04.569769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:04.570719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 89/185] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:04.574593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.629514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.665934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58798963-8d4b-4e8a-a70a-a3f53e15aa26 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.675689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26e6ae3-428f-4099-948d-116ec7cd048c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.699632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.700261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.700566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.700842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.709357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef71132-59e3-4366-a490-d8177f691405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.710008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef71132-59e3-4366-a490-d8177f691405 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.710282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef71132-59e3-4366-a490-d8177f691405 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.710554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef71132-59e3-4366-a490-d8177f691405 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.718487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.721411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef71132-59e3-4366-a490-d8177f691405 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-eef71132-59e3-4366-a490-d8177f691405 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:04.726270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:04.727346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:04.728218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 90/186] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:04.728974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1563e9ca-3112-4879-a836-8478b8fc07e9 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.728974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1563e9ca-3112-4879-a836-8478b8fc07e9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.729435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1563e9ca-3112-4879-a836-8478b8fc07e9 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.730531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 91/187] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:04 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:04.735065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1563e9ca-3112-4879-a836-8478b8fc07e9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.772907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:04.774384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:04.832510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1563e9ca-3112-4879-a836-8478b8fc07e9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.846578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.867858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:04.868486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:04.868759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:04.869031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:04.869754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:04.870057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:04.871490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:04.871577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 97/188] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:04.872389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.880113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 92/189] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:04.880920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:04.919807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:04.969378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a33a4ac-1a26-44ca-81bc-52d3627395c0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:04.979330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85117bd0-9572-4d7a-b100-c6fa84389539 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.002850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830cde6b-143a-4229-8209-deb9157703ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.003588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830cde6b-143a-4229-8209-deb9157703ba None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.003928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830cde6b-143a-4229-8209-deb9157703ba None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.004201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830cde6b-143a-4229-8209-deb9157703ba None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.004920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830cde6b-143a-4229-8209-deb9157703ba None admin] RBAC: Adding query filter params (name=reader) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:05.005219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830cde6b-143a-4229-8209-deb9157703ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.006607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830cde6b-143a-4229-8209-deb9157703ba None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.014184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.014643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.014923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.015226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.015468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 98/190] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.015954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:05.016259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.016713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89069ef7-09f2-4454-a029-28abcfc2bff2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.016996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89069ef7-09f2-4454-a029-28abcfc2bff2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.017270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89069ef7-09f2-4454-a029-28abcfc2bff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.017671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.027762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 93/191] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.031528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.035811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-89069ef7-09f2-4454-a029-28abcfc2bff2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:05.036994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-89069ef7-09f2-4454-a029-28abcfc2bff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:05.072457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.105651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 99/192] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:05.106860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89069ef7-09f2-4454-a029-28abcfc2bff2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.144526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65f8d23-5495-4fd6-8755-0ca8fd677ec7 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.147890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.177203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.177853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.178138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.178414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] PATH_INFO: `/v3/projects/057795b728804191b0733d65e696fe85` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.179601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] RBAC: Authorizing `identity:get_project(project_id=057795b728804191b0733d65e696fe85)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.181068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.182748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 94/193] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/projects/057795b728804191b0733d65e696fe85 => generated 312 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.183538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.209555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89069ef7-09f2-4454-a029-28abcfc2bff2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.243556 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.244207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.244718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.244825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] PATH_INFO: `/v3/users/3c2299c8c5f3491eba84a1a16e32a353` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.258786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2299c8c5f3491eba84a1a16e32a353)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.260294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.261954 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 100/194] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/users/3c2299c8c5f3491eba84a1a16e32a353 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.266716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.285531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c5184-107c-4bdf-9616-eca4e5472336 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.292718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.319561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.320208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.320487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.320762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] PATH_INFO: `/v3/users/ea72826f95f944ff880b7e8905da85d4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.322009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] RBAC: Authorizing `identity:get_user(user_id=ea72826f95f944ff880b7e8905da85d4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.323472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.325249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 95/195] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/users/ea72826f95f944ff880b7e8905da85d4 => generated 289 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.325871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.360215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.364062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f732eb0-edc1-413a-af02-56123427af84 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.397856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.398503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.398779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.399092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] PATH_INFO: `/v3/users/3c2299c8c5f3491eba84a1a16e32a353` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.400234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] RBAC: Authorizing `identity:get_user(user_id=3c2299c8c5f3491eba84a1a16e32a353)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.401675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.403314 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 101/196] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/users/3c2299c8c5f3491eba84a1a16e32a353 => generated 283 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.404123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.428837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6109d69e-af20-41b9-973d-50f6bebeedb1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.464121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.465355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.465355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.465355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.466643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.468148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.469788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 96/197] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.470909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.507854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d79e527-7cc4-4b20-8867-12c6984fe25d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.519350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.542182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.542717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.543021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.543302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.544101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=057795b728804191b0733d65e696fe85, user.id=f7f19a1e144b461c9f801f3d04a0e422) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:05.544413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.545908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.551779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 102/198] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/role_assignments?user.id=f7f19a1e144b461c9f801f3d04a0e422&scope.project.id=057795b728804191b0733d65e696fe85&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:05.552608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.582125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e202b70-641c-45fb-8ccf-aa87e61f92ab None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.588884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.624178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.624832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.625116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.625396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.671312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338a3acc-9e10-40e1-b538-dccaab8531a0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.706007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.708258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.709703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.710350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.710627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.710899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.761697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.795038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:05.796441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:05.823120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 97/199] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:05.823967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.830559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:05.906240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 103/200] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:05.907098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:05.930044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a10f48-be72-4cbf-a98d-6784aadabc6f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:05.966292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:05.966923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:05.967221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:05.967496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:05.979777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:06.008455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534651cf-7e63-4c63-95ca-62f3ce1fe7d3 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:06.036657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:06.042529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.043020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:06.043615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:06.043890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:06.044063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.044186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:06.044875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.046376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.067009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 104/201] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:05 2022] POST /identity/v3/users => generated 297 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:06.079265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:06.177117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 98/202] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:06.178138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:06.195226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d4d0a3-2bd4-465a-86b9-0493c0895cde None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:06.244517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:06.246254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:06.247011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:06.248070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:06.248070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:06.249761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.251740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.254362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 105/203] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:06 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:06.255762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:06.320490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc004de-a9ed-4727-a846-04a5e0a54ef4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:06.321149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:06.368324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:06.368992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:06.369279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:06.369554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:06.371816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.372966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.375048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 99/204] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:06 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:06.376021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:06.385530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61483d21-da6d-4b34-8cde-374171e0586e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:06.423666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:06.424318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:06.424595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:06.424863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:06.426041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.427521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.430235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 106/205] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:06 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:06.430235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:06.502034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e35d65-da69-4c1e-92bc-6e86c1d50183 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:06.518073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:06.551246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:06.551881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:06.552399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:06.552682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:06.553904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=3c2299c8c5f3491eba84a1a16e32a353) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:06.554479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.556196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.560544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec2e131-cf5c-4f61-b507-303b21451bd0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:06.564108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 100/206] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:06 2022] GET /identity/v3/role_assignments?user.id=3c2299c8c5f3491eba84a1a16e32a353&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:06.564954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:06.593972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:06.647847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770c3f19-e777-48a5-977e-2bb6aa350e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:06.648502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c3f19-e777-48a5-977e-2bb6aa350e9a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:06.648787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c3f19-e777-48a5-977e-2bb6aa350e9a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:06.649067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770c3f19-e777-48a5-977e-2bb6aa350e9a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:06.649897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770c3f19-e777-48a5-977e-2bb6aa350e9a None admin] RBAC: Adding query filter params (role.id=7b22f4d466c74886bb0103bf3f7b59d1, scope.project.id=1e52b22e640c409e9307765a7cc712c4, user.id=ea72826f95f944ff880b7e8905da85d4) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:06.650211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770c3f19-e777-48a5-977e-2bb6aa350e9a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.651757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770c3f19-e777-48a5-977e-2bb6aa350e9a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.662130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 107/207] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:06 2022] GET /identity/v3/role_assignments?user.id=ea72826f95f944ff880b7e8905da85d4&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=7b22f4d466c74886bb0103bf3f7b59d1 => generated 608 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:06.698631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634f31aa-a266-4dea-bc95-ec9740528512 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:06.757791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd49849c-f5d1-4b1b-88de-ca8f11260406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:06.758868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd49849c-f5d1-4b1b-88de-ca8f11260406 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:06.759425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd49849c-f5d1-4b1b-88de-ca8f11260406 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:06.759924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd49849c-f5d1-4b1b-88de-ca8f11260406 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:06.760999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd49849c-f5d1-4b1b-88de-ca8f11260406 None admin] RBAC: Adding query filter params (role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=3c2299c8c5f3491eba84a1a16e32a353) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:06.761535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd49849c-f5d1-4b1b-88de-ca8f11260406 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:06.763476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd49849c-f5d1-4b1b-88de-ca8f11260406 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:06.770447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 101/208] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:06 2022] GET /identity/v3/role_assignments?user.id=3c2299c8c5f3491eba84a1a16e32a353&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=629444c0e927466f8da9275a0841de95 => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:08.268665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b43bd4-7593-4e3a-bab9-4637a8969cbe None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.269161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b43bd4-7593-4e3a-bab9-4637a8969cbe None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.269161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b43bd4-7593-4e3a-bab9-4637a8969cbe None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.270660 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 108/209] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:08.276125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98964206-0acd-4be3-b0b5-2e55fbdf7ada None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.276518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98964206-0acd-4be3-b0b5-2e55fbdf7ada None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.276680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98964206-0acd-4be3-b0b5-2e55fbdf7ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.295843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-98964206-0acd-4be3-b0b5-2e55fbdf7ada None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:08.297430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-98964206-0acd-4be3-b0b5-2e55fbdf7ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:08.329099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:08.360567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec46288-c335-4a2b-aac1-5da772c34d3f None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.361198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec46288-c335-4a2b-aac1-5da772c34d3f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.361670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec46288-c335-4a2b-aac1-5da772c34d3f None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.363340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 109/210] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:08.368810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9c41d-ff06-4339-af16-bede8e07b4fa None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.369360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9c41d-ff06-4339-af16-bede8e07b4fa None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.369830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb9c41d-ff06-4339-af16-bede8e07b4fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.383094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 102/211] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:08.389265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9fb9c41d-ff06-4339-af16-bede8e07b4fa None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:08.390793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9fb9c41d-ff06-4339-af16-bede8e07b4fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:08.399480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98964206-0acd-4be3-b0b5-2e55fbdf7ada None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:08.473708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 110/212] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:08.498358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9c41d-ff06-4339-af16-bede8e07b4fa None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:08.510420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98964206-0acd-4be3-b0b5-2e55fbdf7ada None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:08.547520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:08.548093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.548382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.548651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.559472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:08.564526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:08.565461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:08.566221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:08.573502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 103/213] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:08.575047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:08.605046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:08.606324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb9c41d-ff06-4339-af16-bede8e07b4fa None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:08.624411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:08.644710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:08.645619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.646041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.646479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.647874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:08.649595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:08.651524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 111/214] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:08.657474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:08.683290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02270ab9-b2cf-49fe-83f4-60511cc842ce None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:08.718886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:08.719796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.720215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.720982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.721884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:08.722367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:08.724002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:08.733173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 104/215] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:08.739299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:08.764044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfbf73e-2786-41a5-a105-b7e692bceafa None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:08.801608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63950ef-aeb3-44aa-a905-08d5593d56ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:08.802266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63950ef-aeb3-44aa-a905-08d5593d56ca None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.802544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63950ef-aeb3-44aa-a905-08d5593d56ca None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.802818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63950ef-aeb3-44aa-a905-08d5593d56ca None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.803566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63950ef-aeb3-44aa-a905-08d5593d56ca None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:08.805022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63950ef-aeb3-44aa-a905-08d5593d56ca None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:08.828090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 112/216] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:08 2022] POST /identity/v3/users => generated 297 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:57:08.835247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:08.840579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf09b70-a308-444d-af6a-1c1de6e054ee None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.840863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf09b70-a308-444d-af6a-1c1de6e054ee None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.841161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf09b70-a308-444d-af6a-1c1de6e054ee None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.842587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 113/217] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:08.850701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7a8d29-d72b-4125-838c-aebe65bb84dc None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:08.852470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff8c35-312a-44d4-b159-b78a343d2156 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.852756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff8c35-312a-44d4-b159-b78a343d2156 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.853035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bff8c35-312a-44d4-b159-b78a343d2156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.873145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6bff8c35-312a-44d4-b159-b78a343d2156 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:08.874381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6bff8c35-312a-44d4-b159-b78a343d2156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:08.903291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b8c166-6a70-4928-8bdf-c5cbe8a2476d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:08.904043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b8c166-6a70-4928-8bdf-c5cbe8a2476d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.904327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b8c166-6a70-4928-8bdf-c5cbe8a2476d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.904605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b8c166-6a70-4928-8bdf-c5cbe8a2476d None admin] PATH_INFO: `/v3/users/0ed828da986d4a4aa1ed20abceb0b13a` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.916018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:08.920044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b8c166-6a70-4928-8bdf-c5cbe8a2476d None admin] RBAC: Authorizing `identity:get_user(user_id=0ed828da986d4a4aa1ed20abceb0b13a)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:08.921555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b8c166-6a70-4928-8bdf-c5cbe8a2476d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:08.923782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 105/218] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:08 2022] GET /identity/v3/users/0ed828da986d4a4aa1ed20abceb0b13a => generated 297 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:08.925088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd585ad-6b4e-4046-835e-67dbee5dbbb3 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.925376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd585ad-6b4e-4046-835e-67dbee5dbbb3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.925657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd585ad-6b4e-4046-835e-67dbee5dbbb3 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.926982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 106/219] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:08.932185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd585ad-6b4e-4046-835e-67dbee5dbbb3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:08.960898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 114/220] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:08.962295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206fa82a-7f1c-4062-b339-fdd79c303c8c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:08.962584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206fa82a-7f1c-4062-b339-fdd79c303c8c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:08.962860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206fa82a-7f1c-4062-b339-fdd79c303c8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:08.992143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-206fa82a-7f1c-4062-b339-fdd79c303c8c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:08.993370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-206fa82a-7f1c-4062-b339-fdd79c303c8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:09.063178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd585ad-6b4e-4046-835e-67dbee5dbbb3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.075973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 115/221] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:09.079138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206fa82a-7f1c-4062-b339-fdd79c303c8c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.103660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.104538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.104994 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.105267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.131001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:09.203132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:09.205110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:09.206583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:09.210271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206fa82a-7f1c-4062-b339-fdd79c303c8c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.256828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.258034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.258827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.259294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.273795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:09.281919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:09.282802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:09.283622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:09.286652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 116/222] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:09.287664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.320489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 107/223] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:09.321725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.374640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:09.428003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3f50fd-7a47-4106-8894-65ba231a4607 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.428415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aefd7f3-0a42-4a8d-b568-55aeead95e12 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.465442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.466228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.466823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.467244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.468048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:09.468453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:09.469923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:09.474685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.475600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.476042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.477032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.480257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 108/224] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:09.481200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.490599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:09.497340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:09.498345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:09.499142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:09.500449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 117/225] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:09.501998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.530659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:09.606136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8ace4b-b5c2-4aed-a7d1-b1a327ece141 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.619495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:09.621636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e4133-29c7-43e9-ace2-dd6bdb479b69 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.651560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.652805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.653551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.653994 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.656449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:09.659136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:09.662121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 109/226] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:09.663108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.669235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.670025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.670400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.670772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] PATH_INFO: `/v3/users/3c2299c8c5f3491eba84a1a16e32a353` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.672166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2299c8c5f3491eba84a1a16e32a353)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:09.674184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:09.675979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 118/227] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/users/3c2299c8c5f3491eba84a1a16e32a353 => generated 283 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:09.676965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.773341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:09.789855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8ec578-1b48-4780-9e25-3d01ce0811c0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.799167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8c07d9-7727-42d6-bd38-c627acb9c3e8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:09.827331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.827964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.828244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.828518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.829396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=0ed828da986d4a4aa1ed20abceb0b13a) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:09.829621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:09.831216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:09.836827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 119/228] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/role_assignments?user.id=0ed828da986d4a4aa1ed20abceb0b13a&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:09.843914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.844686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:09.845904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:09.846370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:09.846853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:09.847829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:09.848375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:09.850327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:09.861875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 110/229] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:09.873102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:09.954502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:10.030443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4aa4d8-e580-467d-a93f-db3532f95565 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:10.071502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:10.072169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:10.072448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:10.072720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] PATH_INFO: `/v3/users/3c2299c8c5f3491eba84a1a16e32a353` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:10.074626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2299c8c5f3491eba84a1a16e32a353)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:10.076838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:10.078764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 111/230] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/users/3c2299c8c5f3491eba84a1a16e32a353 => generated 283 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:10.079759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4251481-0052-4c30-9409-96e3a6d29713 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:10.088001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:10.123144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:10.124169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:10.124671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:10.125219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:10.126654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:10.128357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:10.131106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 120/231] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:09 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:10.138163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:10.187340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:10.272760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6db4104-9183-4434-8efa-90c7a95f0598 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:10.283378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:10.316763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:10.318275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:10.318565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:10.318837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:10.325566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:10.327097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:10.328981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 112/232] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:10 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:10.334646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:10.376571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f4405-59a0-40b9-ad53-25426d11b539 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:10.420265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03505cd8-ec36-44e3-8ad8-a1bda10668fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:10.420967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03505cd8-ec36-44e3-8ad8-a1bda10668fa None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:10.421249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03505cd8-ec36-44e3-8ad8-a1bda10668fa None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:10.421570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03505cd8-ec36-44e3-8ad8-a1bda10668fa None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/3c2299c8c5f3491eba84a1a16e32a353/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:10.428873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03505cd8-ec36-44e3-8ad8-a1bda10668fa None admin] RBAC: Authorizing `identity:create_grant(project_id=1f9b890871df4d1f96f60a844ee6b347, user_id=3c2299c8c5f3491eba84a1a16e32a353, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:10.430553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03505cd8-ec36-44e3-8ad8-a1bda10668fa None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:10.452559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 121/233] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:10 2022] PUT /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/3c2299c8c5f3491eba84a1a16e32a353/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:10.495252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec76431-c1e1-48b4-97f8-4fc434576453 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:10.545409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98f2d74-7409-408b-912a-96f98d0dfc7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:10.546102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98f2d74-7409-408b-912a-96f98d0dfc7c None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:10.546373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98f2d74-7409-408b-912a-96f98d0dfc7c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:10.546645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98f2d74-7409-408b-912a-96f98d0dfc7c None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/3c2299c8c5f3491eba84a1a16e32a353/roles/629444c0e927466f8da9275a0841de95` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:10.548945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98f2d74-7409-408b-912a-96f98d0dfc7c None admin] RBAC: Authorizing `identity:create_grant(project_id=1f9b890871df4d1f96f60a844ee6b347, user_id=3c2299c8c5f3491eba84a1a16e32a353, role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:10.550445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98f2d74-7409-408b-912a-96f98d0dfc7c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:10.578628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 113/234] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:10 2022] PUT /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/3c2299c8c5f3491eba84a1a16e32a353/roles/629444c0e927466f8da9275a0841de95 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:11.126157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e937c-175f-420c-a608-0ebe0c145eeb None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.126550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e937c-175f-420c-a608-0ebe0c145eeb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.126738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e937c-175f-420c-a608-0ebe0c145eeb None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.128344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 122/235] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:11.133987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d16109-60ce-455e-94ec-3f924a996194 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.134272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d16109-60ce-455e-94ec-3f924a996194 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.134549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d16109-60ce-455e-94ec-3f924a996194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.153495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e380fa-215d-46f4-a9ed-1f034cee77b1 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.153803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e380fa-215d-46f4-a9ed-1f034cee77b1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.154042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e380fa-215d-46f4-a9ed-1f034cee77b1 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.155576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 123/236] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:11.157813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-80d16109-60ce-455e-94ec-3f924a996194 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:11.159061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-80d16109-60ce-455e-94ec-3f924a996194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:11.161357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90417b1-611d-44f5-b951-fa6fbabc6f1d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.161634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90417b1-611d-44f5-b951-fa6fbabc6f1d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.161908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90417b1-611d-44f5-b951-fa6fbabc6f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.188010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f90417b1-611d-44f5-b951-fa6fbabc6f1d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:11.189246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f90417b1-611d-44f5-b951-fa6fbabc6f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:11.209823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:11.273847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 114/237] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:11.285201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 124/238] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:11.297820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d16109-60ce-455e-94ec-3f924a996194 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:11.340754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90417b1-611d-44f5-b951-fa6fbabc6f1d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:11.361781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:11.401589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d16109-60ce-455e-94ec-3f924a996194 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:11.437724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:11.438391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.438669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.438958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.440112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:11.441568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:11.443560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 115/239] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:11.453011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:11.489806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:11.558724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90417b1-611d-44f5-b951-fa6fbabc6f1d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:11.563043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f009cf-1967-4b9f-b556-55c8c4f0844f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:11.603421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:11.604181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.604461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.605591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.606313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:11.607863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:11.627009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:11.627680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.627960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.628233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.649870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:11.658699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:11.661948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:11.664479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 125/240] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:11.665910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 116/241] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:11 2022] POST /identity/v3/users => generated 293 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:11.676114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:11.791732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:11.839035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dafb16-7bfc-45e9-a9c1-79e058eabc30 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:11.886109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:11.886753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:11.887052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:11.887327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:11.888233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] RBAC: Adding query filter params (name=reader) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:11.888494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:11.890009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:11.902691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 126/242] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:11.909071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.028722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:12.045902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7916007-0f29-4206-8a5b-ba00132dcd05 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:12.093084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3082985-2b1d-4aca-90b4-0dd97c522cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:12.093672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3082985-2b1d-4aca-90b4-0dd97c522cbf None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.093951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3082985-2b1d-4aca-90b4-0dd97c522cbf None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.094227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3082985-2b1d-4aca-90b4-0dd97c522cbf None admin] PATH_INFO: `/v3/users/036b8d9607994e19a57f546ebdfc5e9d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.114092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3082985-2b1d-4aca-90b4-0dd97c522cbf None admin] RBAC: Authorizing `identity:get_user(user_id=036b8d9607994e19a57f546ebdfc5e9d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:12.115617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3082985-2b1d-4aca-90b4-0dd97c522cbf None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:12.117344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 117/243] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:11 2022] GET /identity/v3/users/036b8d9607994e19a57f546ebdfc5e9d => generated 297 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:12.123198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.207116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6de33c-e809-456e-b9a8-7e8aa397e5fd None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.207396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6de33c-e809-456e-b9a8-7e8aa397e5fd None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.207683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6de33c-e809-456e-b9a8-7e8aa397e5fd None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.209202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 118/244] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:12.214696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c75b8-15f7-4335-916c-73a543a886a9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.215412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c75b8-15f7-4335-916c-73a543a886a9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.216243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c75b8-15f7-4335-916c-73a543a886a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.241366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-894c75b8-15f7-4335-916c-73a543a886a9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:12.242834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-894c75b8-15f7-4335-916c-73a543a886a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:12.252061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4037a2a-debd-4821-b2b2-5f1fc7dc549c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:12.281998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:12.288381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:12.289508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.289794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.290074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.332119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 119/245] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:12.355714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894c75b8-15f7-4335-916c-73a543a886a9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.370860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:12.372551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:12.458784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:12.487714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894c75b8-15f7-4335-916c-73a543a886a9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:12.507974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 127/246] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:12.513149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.528026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:12.528715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.528988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.529264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.541927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:12.547152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:12.548110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:12.549039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 120/247] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:12.555239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.572245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:12.680800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c56d24-ab5b-48fb-92a5-39dd54f57dc8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:12.717784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51f89b5-4ec9-4398-9afa-3d959f2c1501 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:12.751361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:12.752119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.752397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.752668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.753562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:12.754240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.754524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.754796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.755567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:12.755871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:12.757286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:12.759425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:12.761199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:12.762971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:12.764946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 128/248] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:12 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:12.770870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 121/249] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:12.772292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.772721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.876121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b0709-34f3-442c-abe4-8394b4caf92b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:12.915135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:12.915797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:12.916074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:12.916353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] PATH_INFO: `/v3/users/0ed828da986d4a4aa1ed20abceb0b13a` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:12.918055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] RBAC: Authorizing `identity:get_user(user_id=0ed828da986d4a4aa1ed20abceb0b13a)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:12.919593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:12.921395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 122/250] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:12 2022] GET /identity/v3/users/0ed828da986d4a4aa1ed20abceb0b13a => generated 297 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:12.928504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:12.958831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:12.961488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4506ed5d-1a81-4fc8-b0be-9a91e7de94fd None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:13.071947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:13.072585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.072860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.073136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.073943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] RBAC: Adding query filter params (role.id=7b22f4d466c74886bb0103bf3f7b59d1, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=036b8d9607994e19a57f546ebdfc5e9d) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:13.074247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:13.075801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:13.090670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 129/251] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:12 2022] GET /identity/v3/role_assignments?user.id=036b8d9607994e19a57f546ebdfc5e9d&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=7b22f4d466c74886bb0103bf3f7b59d1 => generated 259 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:13.107430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:13.167456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a1fc03-88aa-486f-ad3e-af569ce9a7dd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:13.217406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d2e8fe-7dee-4c3c-bcb5-fca9106fe3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:13.217968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d2e8fe-7dee-4c3c-bcb5-fca9106fe3db None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.218245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d2e8fe-7dee-4c3c-bcb5-fca9106fe3db None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.218516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d2e8fe-7dee-4c3c-bcb5-fca9106fe3db None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.223774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d2e8fe-7dee-4c3c-bcb5-fca9106fe3db None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:13.225281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d2e8fe-7dee-4c3c-bcb5-fca9106fe3db None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:13.227076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 123/252] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:12 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:13.233316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:13.352628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324464fd-097c-4915-965b-ed9f67679e66 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.353282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324464fd-097c-4915-965b-ed9f67679e66 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.353759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324464fd-097c-4915-965b-ed9f67679e66 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.355528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 124/253] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:13.361567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b37aac1-b3a9-463e-8d07-24166e3e51ee None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.362242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b37aac1-b3a9-463e-8d07-24166e3e51ee None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.362707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b37aac1-b3a9-463e-8d07-24166e3e51ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.388405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5b37aac1-b3a9-463e-8d07-24166e3e51ee None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:13.391500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5b37aac1-b3a9-463e-8d07-24166e3e51ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:13.397195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f72fc92-73ef-49de-b963-0767f593b265 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:13.454906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:13.457287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.457819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.458433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/0ed828da986d4a4aa1ed20abceb0b13a/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.461654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] RBAC: Authorizing `identity:create_grant(project_id=1f9b890871df4d1f96f60a844ee6b347, user_id=0ed828da986d4a4aa1ed20abceb0b13a, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:13.463516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:13.491506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 130/254] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:13 2022] PUT /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/0ed828da986d4a4aa1ed20abceb0b13a/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:13.500765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 125/255] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:13.527070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:13.586069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:13.587347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e23e8d-4d74-4ae9-837b-262714d018fe None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.587634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e23e8d-4d74-4ae9-837b-262714d018fe None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.587904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e23e8d-4d74-4ae9-837b-262714d018fe None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.589532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 126/256] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:13.594803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe2eb3e-ddd0-40ef-b9ad-4398521ec7df None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.595126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe2eb3e-ddd0-40ef-b9ad-4398521ec7df None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.595843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe2eb3e-ddd0-40ef-b9ad-4398521ec7df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.617737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0fe2eb3e-ddd0-40ef-b9ad-4398521ec7df None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:13.619483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0fe2eb3e-ddd0-40ef-b9ad-4398521ec7df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:13.711818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949b410a-d043-42e9-8064-0b4513be5cb6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:13.758539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:13.775133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 127/257] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:13.789465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:13.790133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:13.790410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:13.790682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:13.811074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:13.816533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:13.817422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:13.818377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 131/258] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:13.824106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe2eb3e-ddd0-40ef-b9ad-4398521ec7df None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:13.829357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:14.000632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:14.010938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe2eb3e-ddd0-40ef-b9ad-4398521ec7df None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:14.032337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27a51c4-6777-44e9-9f7c-4377533766fa None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:14.075097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:14.075911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:14.076186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:14.076455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:14.077182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:14.077483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:14.078865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:14.091807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 128/259] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:14.097565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:14.098076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:14.098996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:14.099484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:14.099941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:14.123084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:14.128741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:14.129599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:14.130449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:14.131534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 132/260] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:14.142862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:14.199308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:14.278764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1109b4-6706-42cf-b2e1-68e8a4014dd4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:14.328137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:14.329214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:14.329685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:14.330146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] PATH_INFO: `/v3/users/3c2299c8c5f3491eba84a1a16e32a353` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:14.335870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2299c8c5f3491eba84a1a16e32a353)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:14.337595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:14.340057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 129/261] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:14 2022] GET /identity/v3/users/3c2299c8c5f3491eba84a1a16e32a353 => generated 283 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:14.347598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:14.355685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bed536-61bb-4faf-91dd-42ed0457135b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:14.480357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7197a3aa-e1eb-4450-86f6-1c872876df78 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:14.500598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:14.501241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:14.501518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:14.501789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:14.502524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:14.502827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:14.504261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:14.510760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:14.525825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 133/262] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 383 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:14.526915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:14.528063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:14.528609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:14.529122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:14.532305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:14.544581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:14.692034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:14.693530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:14.736390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d858e8-cad1-4305-afb4-064c50120585 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:14.800421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:14.801018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:14.801569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:14.801847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] PATH_INFO: `/v3/users/3c2299c8c5f3491eba84a1a16e32a353` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:14.803320 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] RBAC: Authorizing `identity:get_user(user_id=3c2299c8c5f3491eba84a1a16e32a353)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:14.804870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:14.806582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 134/263] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:14 2022] GET /identity/v3/users/3c2299c8c5f3491eba84a1a16e32a353 => generated 283 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:14.812925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:14.927886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 130/264] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 581 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:57:14.932992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:14.944327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:14.985937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0ee992-8c56-4ee4-9be6-078699b60a35 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:15.004750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:15.043550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4e40d2-6ecc-45e6-9cd1-8d1ddc3be180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:15.044219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4e40d2-6ecc-45e6-9cd1-8d1ddc3be180 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.044495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4e40d2-6ecc-45e6-9cd1-8d1ddc3be180 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.044762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4e40d2-6ecc-45e6-9cd1-8d1ddc3be180 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.130111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1878520-8d9b-4a5a-a50f-f2440618927f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:15.162980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4e40d2-6ecc-45e6-9cd1-8d1ddc3be180 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:15.164475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4e40d2-6ecc-45e6-9cd1-8d1ddc3be180 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:15.169098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404b0c23-098f-484b-b452-4757ced95963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:15.169759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404b0c23-098f-484b-b452-4757ced95963 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.170034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404b0c23-098f-484b-b452-4757ced95963 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.170304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404b0c23-098f-484b-b452-4757ced95963 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.171512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404b0c23-098f-484b-b452-4757ced95963 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:15.173081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404b0c23-098f-484b-b452-4757ced95963 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:15.175694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 131/265] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:14 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:15.176778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82068880-1056-4101-8320-cfc270124446 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.177062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82068880-1056-4101-8320-cfc270124446 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.177332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82068880-1056-4101-8320-cfc270124446 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.183456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 132/266] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:15.185029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc61ca16-7fd6-48c8-8d09-0f44a318bb0a None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.185310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc61ca16-7fd6-48c8-8d09-0f44a318bb0a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.185582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc61ca16-7fd6-48c8-8d09-0f44a318bb0a None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.187247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 133/267] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:15.188976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc61ca16-7fd6-48c8-8d09-0f44a318bb0a None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:15.257153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:15.283683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 135/268] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:57:15.285039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527b437-2cd2-45d3-99a6-2283c9be4713 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.285327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527b437-2cd2-45d3-99a6-2283c9be4713 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.285598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6527b437-2cd2-45d3-99a6-2283c9be4713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.297930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:15.305485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6527b437-2cd2-45d3-99a6-2283c9be4713 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:15.306863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc61ca16-7fd6-48c8-8d09-0f44a318bb0a None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:15.306992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6527b437-2cd2-45d3-99a6-2283c9be4713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:15.344377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330487c4-5f8f-445d-a8d8-b335e4f6dcf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:15.345514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330487c4-5f8f-445d-a8d8-b335e4f6dcf2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.345882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330487c4-5f8f-445d-a8d8-b335e4f6dcf2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.346247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330487c4-5f8f-445d-a8d8-b335e4f6dcf2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.347211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330487c4-5f8f-445d-a8d8-b335e4f6dcf2 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=3c2299c8c5f3491eba84a1a16e32a353) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:15.347635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330487c4-5f8f-445d-a8d8-b335e4f6dcf2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:15.349166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330487c4-5f8f-445d-a8d8-b335e4f6dcf2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:15.363266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 134/269] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:15 2022] GET /identity/v3/role_assignments?user.id=3c2299c8c5f3491eba84a1a16e32a353&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:15.364901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21c2b70-88a8-440e-802a-7f1a8b2741f8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.365426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21c2b70-88a8-440e-802a-7f1a8b2741f8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.365886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21c2b70-88a8-440e-802a-7f1a8b2741f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.382112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 136/270] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:15.383305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6527b437-2cd2-45d3-99a6-2283c9be4713 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:15.404309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f21c2b70-88a8-440e-802a-7f1a8b2741f8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:15.405725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f21c2b70-88a8-440e-802a-7f1a8b2741f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:15.488179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 135/271] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:15.489398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21c2b70-88a8-440e-802a-7f1a8b2741f8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:15.539590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6527b437-2cd2-45d3-99a6-2283c9be4713 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:15.542197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:15.587306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:15.602883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21c2b70-88a8-440e-802a-7f1a8b2741f8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:15.605411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:15.606136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.606441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.606912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.608361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:15.609963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:15.611945 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 137/272] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:15 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:15.612892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:15.643659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:15.644675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.644840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.645208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:15.663007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:15.664017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:15.664992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 136/273] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:15.665728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:15.778195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e8be43-3914-45a0-9579-343edca37c9c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:15.823442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:15.824382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.824921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.825360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.827215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:15.829490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:15.832129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 138/274] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:15.833060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:15.840007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f0d4fd-05cc-465b-9c32-2897420991f8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:15.881344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:15.882318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:15.882810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:15.883317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:15.884330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] RBAC: Adding query filter params (role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=3c2299c8c5f3491eba84a1a16e32a353) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:15.884829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:15.886526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:15.889669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:15.894129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 137/275] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:15 2022] GET /identity/v3/role_assignments?user.id=3c2299c8c5f3491eba84a1a16e32a353&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=629444c0e927466f8da9275a0841de95 => generated 608 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:15.895311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:15.921038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:15.962936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb06a24-42b9-48e2-a455-e7583a15c5bd None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.028426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b2f37b-cba6-4e58-98ef-93451fe58c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.028426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b2f37b-cba6-4e58-98ef-93451fe58c9e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.028426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b2f37b-cba6-4e58-98ef-93451fe58c9e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.028426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b2f37b-cba6-4e58-98ef-93451fe58c9e None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.028970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b2f37b-cba6-4e58-98ef-93451fe58c9e None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:16.029209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b2f37b-cba6-4e58-98ef-93451fe58c9e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:16.030653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9b2f37b-cba6-4e58-98ef-93451fe58c9e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:16.045027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 139/276] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:16.046728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8069c65-2dd0-44b2-a9a3-038e595e12d4 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.047193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8069c65-2dd0-44b2-a9a3-038e595e12d4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.047622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8069c65-2dd0-44b2-a9a3-038e595e12d4 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.048656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c85c5c-65e2-494a-a814-c317934d5a87 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.049473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 140/277] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:16.056835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ef052-1a98-4617-b76a-aa7c4dab2e47 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.057145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ef052-1a98-4617-b76a-aa7c4dab2e47 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.057520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ef052-1a98-4617-b76a-aa7c4dab2e47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.077650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9b8ef052-1a98-4617-b76a-aa7c4dab2e47 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:16.079110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9b8ef052-1a98-4617-b76a-aa7c4dab2e47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:16.091649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.092716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.093266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.093727 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.094926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:16.097063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:16.123922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 138/278] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:15 2022] POST /identity/v3/users => generated 293 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:16.127003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.159959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 141/279] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:16.192100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8ef052-1a98-4617-b76a-aa7c4dab2e47 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.243204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:16.245587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:16.291815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d12319-3088-4a11-bcc6-fb097144f249 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.343542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52c1033-0490-41be-96f4-891c7edf1ce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.344303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8ef052-1a98-4617-b76a-aa7c4dab2e47 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.344775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52c1033-0490-41be-96f4-891c7edf1ce3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.345601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52c1033-0490-41be-96f4-891c7edf1ce3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.346176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52c1033-0490-41be-96f4-891c7edf1ce3 None admin] PATH_INFO: `/v3/users/e2dc4cf3197f481daf10790ae1a9f50d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.361295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52c1033-0490-41be-96f4-891c7edf1ce3 None admin] RBAC: Authorizing `identity:get_user(user_id=e2dc4cf3197f481daf10790ae1a9f50d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:16.363067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52c1033-0490-41be-96f4-891c7edf1ce3 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:16.365869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 139/280] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/users/e2dc4cf3197f481daf10790ae1a9f50d => generated 293 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:16.366762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727832a5-9227-4f8f-8073-790720df0a15 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.367075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727832a5-9227-4f8f-8073-790720df0a15 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.367356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727832a5-9227-4f8f-8073-790720df0a15 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.368756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 140/281] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:16.372384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727832a5-9227-4f8f-8073-790720df0a15 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.382233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae1b09-9528-4b0b-9dcd-aef635f9a938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.382899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae1b09-9528-4b0b-9dcd-aef635f9a938 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.383209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae1b09-9528-4b0b-9dcd-aef635f9a938 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.383482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae1b09-9528-4b0b-9dcd-aef635f9a938 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.396560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ae1b09-9528-4b0b-9dcd-aef635f9a938 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-94ae1b09-9528-4b0b-9dcd-aef635f9a938 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:16.401797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:16.402687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:16.403584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 142/282] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:16.404584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c69b94e-6024-4324-956e-64df030d3fa5 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.404897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c69b94e-6024-4324-956e-64df030d3fa5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.405137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c69b94e-6024-4324-956e-64df030d3fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.423544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4c69b94e-6024-4324-956e-64df030d3fa5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:16.424696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4c69b94e-6024-4324-956e-64df030d3fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:16.499893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727832a5-9227-4f8f-8073-790720df0a15 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.506238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 143/283] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:16.507388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c69b94e-6024-4324-956e-64df030d3fa5 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.531601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:16.537002 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:16.548404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.548601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.549026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.549304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.551007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, user.id=e2dc4cf3197f481daf10790ae1a9f50d, scope.system=all) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:16.553792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:16.553792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:16.561923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 141/284] 10.222.0.42 () {56 vars in 1308 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/role_assignments?user.id=e2dc4cf3197f481daf10790ae1a9f50d&scope.system=all&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:16.563023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.635731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c69b94e-6024-4324-956e-64df030d3fa5 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.693634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.694218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.694508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.694786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.695584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:16.695895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:16.697378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:16.701311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74728a9-e24a-4811-b4c1-f8ffa9ff92e5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.719158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 144/285] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:16.728102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.798495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.799837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.800123 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.800399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.821022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:16.826769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:16.827657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:16.828455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:16.835414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:16.837854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 142/286] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:16.844033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.844883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb6593e-cf17-40f7-aed2-a5c542917759 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:16.914701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:16.915736 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:16.916185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:16.916627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] PATH_INFO: `/v3/users/036b8d9607994e19a57f546ebdfc5e9d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:16.918189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] RBAC: Authorizing `identity:get_user(user_id=036b8d9607994e19a57f546ebdfc5e9d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:16.919958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:16.924285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 145/287] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/users/036b8d9607994e19a57f546ebdfc5e9d => generated 297 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:16.930388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:16.942530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:17.010641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4362c0e7-ff91-4d54-85a2-590d11bf4300 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:17.094411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:17.095429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:17.095714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:17.095984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:17.096779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:17.097081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:17.100312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:17.123164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 143/288] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:17.132854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d61bfa-1df2-4908-8670-563f4227bc06 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:17.140982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:17.188744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:17.189409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:17.189691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:17.189969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:17.191255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:17.192751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:17.194446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 146/289] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:16 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:17.201615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:17.251313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:17.307197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f4476f-043e-4d34-8c97-09a640d1d699 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:17.320580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:17.328642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbac0871-6a3c-49b6-939f-0e3365376cea None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:17.342882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:17.343583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:17.343869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:17.344150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] PATH_INFO: `/v3/users/0ed828da986d4a4aa1ed20abceb0b13a` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:17.345733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] RBAC: Authorizing `identity:get_user(user_id=0ed828da986d4a4aa1ed20abceb0b13a)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:17.347260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:17.353534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 144/290] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:17 2022] GET /identity/v3/users/0ed828da986d4a4aa1ed20abceb0b13a => generated 297 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:17.365015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:17.375383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:17.376365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:17.376839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:17.377359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/036b8d9607994e19a57f546ebdfc5e9d/roles/7b22f4d466c74886bb0103bf3f7b59d1` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:17.379579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] RBAC: Authorizing `identity:create_grant(project_id=1f9b890871df4d1f96f60a844ee6b347, user_id=036b8d9607994e19a57f546ebdfc5e9d, role_id=7b22f4d466c74886bb0103bf3f7b59d1)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:17.381276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:17.403013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 147/291] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:17 2022] PUT /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347/users/036b8d9607994e19a57f546ebdfc5e9d/roles/7b22f4d466c74886bb0103bf3f7b59d1 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:17.510987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3534f87f-0423-444f-ac37-349f6586f57a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:17.556061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:17.557097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:17.557645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:17.558144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:17.632119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:17.660914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:17.663105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:17.798458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 145/292] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:17.805092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:17.892225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:17.912667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cce007f-a566-4dd2-8cdb-4147b83d2401 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:17.949859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad271c5-f800-4cdd-aed9-f9735bfa5be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:17.950645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad271c5-f800-4cdd-aed9-f9735bfa5be9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:17.950927 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad271c5-f800-4cdd-aed9-f9735bfa5be9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:17.951234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad271c5-f800-4cdd-aed9-f9735bfa5be9 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:17.952466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad271c5-f800-4cdd-aed9-f9735bfa5be9 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:17.954047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad271c5-f800-4cdd-aed9-f9735bfa5be9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:17.960797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 148/293] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:17 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:17.962239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:18.075004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:18.104821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7b4eac-0197-4d25-a5f2-99727caccf42 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:18.149699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee87125-bbe0-49a8-9a90-4f383bc4b8f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:18.150248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee87125-bbe0-49a8-9a90-4f383bc4b8f0 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:18.150531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee87125-bbe0-49a8-9a90-4f383bc4b8f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:18.150809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee87125-bbe0-49a8-9a90-4f383bc4b8f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:18.151643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee87125-bbe0-49a8-9a90-4f383bc4b8f0 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=0ed828da986d4a4aa1ed20abceb0b13a) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:18.151954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee87125-bbe0-49a8-9a90-4f383bc4b8f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:18.153491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee87125-bbe0-49a8-9a90-4f383bc4b8f0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:18.163677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 146/294] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:17 2022] GET /identity/v3/role_assignments?user.id=0ed828da986d4a4aa1ed20abceb0b13a&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 608 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:18.719531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d05df-b153-46a8-87cb-4e8e7fe861ea None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:18.719932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d05df-b153-46a8-87cb-4e8e7fe861ea None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:18.720466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d05df-b153-46a8-87cb-4e8e7fe861ea None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:18.722152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 149/295] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:18.727579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7138c2-a5ef-401d-9d77-8bbb6f6482ca None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:18.727871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7138c2-a5ef-401d-9d77-8bbb6f6482ca None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:18.728153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7138c2-a5ef-401d-9d77-8bbb6f6482ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:18.748626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-fe7138c2-a5ef-401d-9d77-8bbb6f6482ca None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:18.749849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fe7138c2-a5ef-401d-9d77-8bbb6f6482ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:18.789922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b830e1-a817-498c-bf10-6f779c9c040e None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:18.790226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b830e1-a817-498c-bf10-6f779c9c040e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:18.790489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b830e1-a817-498c-bf10-6f779c9c040e None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:18.792043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 150/296] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:18.798603 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a68885-9d46-413b-a081-7f88a2d2579e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:18.798892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a68885-9d46-413b-a081-7f88a2d2579e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:18.799202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a68885-9d46-413b-a081-7f88a2d2579e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:18.821138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-28a68885-9d46-413b-a081-7f88a2d2579e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:18.822826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-28a68885-9d46-413b-a081-7f88a2d2579e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:18.834295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 147/297] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:18.857716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7138c2-a5ef-401d-9d77-8bbb6f6482ca None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:18.904616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 151/298] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:18.905809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04f68c-f8aa-4c4f-9a2c-308a28e60c1e None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:18.906590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04f68c-f8aa-4c4f-9a2c-308a28e60c1e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:18.906590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e04f68c-f8aa-4c4f-9a2c-308a28e60c1e None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:18.915407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 152/299] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:18 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:18.923205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f384d119-62b5-4145-a079-5e873b9980c9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:18.923624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f384d119-62b5-4145-a079-5e873b9980c9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:18.923991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f384d119-62b5-4145-a079-5e873b9980c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:18.942333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f384d119-62b5-4145-a079-5e873b9980c9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:18.942529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:18.943754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f384d119-62b5-4145-a079-5e873b9980c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:18.983993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7138c2-a5ef-401d-9d77-8bbb6f6482ca None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.019624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 153/300] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:19.020632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f384d119-62b5-4145-a079-5e873b9980c9 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.036992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.037743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.038091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.038432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.042218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:19.050125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:19.056710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:19.057571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:19.058360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:19.058966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 148/301] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:19.066997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.140073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f384d119-62b5-4145-a079-5e873b9980c9 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.170442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24542544-0fb6-4b9e-b187-c05dcc171aa5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.176895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.177552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.177980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.178084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.191385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:19.196497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:19.197407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:19.198221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 154/302] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:19.198866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.213940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.214595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.214934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.215237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.216424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.217884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.219789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 149/303] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:19.220918 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.223624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:19.277204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:19.312722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38daca2-c0cf-4e08-a354-409f3c2cc430 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.325341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db193ead-20a6-41fa-b439-71e9d9b3bec3 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.351422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.352382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.352881 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.353342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.354378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:19.354903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.356407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.360842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.361472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.361759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.362033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.362760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:19.363142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.364513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.370473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 155/304] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:19.371382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.378445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 150/305] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:19.378445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.444625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:19.483349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a18ac48-df93-4f76-9e72-0fb2829aa2c7 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.486223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a485b9ac-90dc-4b0f-ba12-14b4262ed91d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.513303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:19.520352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.520995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.521272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.521554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] PATH_INFO: `/v3/groups` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.522224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.522322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.522963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.523317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.523590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] PATH_INFO: `/v3/users/d78e6e040620489b8456d6d33062feae` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.523735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.539724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] RBAC: Authorizing `identity:get_user(user_id=d78e6e040620489b8456d6d33062feae)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.541117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.541363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 156/306] 10.222.0.42 () {60 vars in 1134 bytes} [Tue Jul 26 02:57:19 2022] POST /identity/v3/groups => generated 229 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:19.547393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.547490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 151/307] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/users/d78e6e040620489b8456d6d33062feae => generated 293 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:19.556325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.679262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd0a04f-1f89-4ce7-90fd-5d7df411b43f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.703534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd2624-0ed1-47d6-bac2-903ce393344d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.718984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.719962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.720803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.721244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] PATH_INFO: `/v3/users/e2dc4cf3197f481daf10790ae1a9f50d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.722630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] RBAC: Authorizing `identity:get_user(user_id=e2dc4cf3197f481daf10790ae1a9f50d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.724340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.731008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 157/308] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/users/e2dc4cf3197f481daf10790ae1a9f50d => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:19.732433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:19.733475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:19.752139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb690be-7d81-41b7-a4b6-0fe27506f0b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.753032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb690be-7d81-41b7-a4b6-0fe27506f0b1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.753309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb690be-7d81-41b7-a4b6-0fe27506f0b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.753578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb690be-7d81-41b7-a4b6-0fe27506f0b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.754645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb690be-7d81-41b7-a4b6-0fe27506f0b1 None admin] RBAC: Adding query filter params (role.id=7b22f4d466c74886bb0103bf3f7b59d1, user.id=d78e6e040620489b8456d6d33062feae, scope.system=all) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:19.755356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb690be-7d81-41b7-a4b6-0fe27506f0b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.756827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb690be-7d81-41b7-a4b6-0fe27506f0b1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.766985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 152/309] 10.222.0.42 () {56 vars in 1308 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/role_assignments?user.id=d78e6e040620489b8456d6d33062feae&scope.system=all&role.id=7b22f4d466c74886bb0103bf3f7b59d1 => generated 226 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:19.810773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:19.853092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368182c8-f544-439f-bd81-8f63c8a091df None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.853651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368182c8-f544-439f-bd81-8f63c8a091df None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.853651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368182c8-f544-439f-bd81-8f63c8a091df None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.855370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 153/310] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:19.861035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a918c-bad8-46ce-a627-add4af1eaef4 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.861342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a918c-bad8-46ce-a627-add4af1eaef4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.861595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4a918c-bad8-46ce-a627-add4af1eaef4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.867830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99bd4dd-e295-4ad6-b56b-474937dc1103 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:19.881008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-fc4a918c-bad8-46ce-a627-add4af1eaef4 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:19.882330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fc4a918c-bad8-46ce-a627-add4af1eaef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:19.915191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:19.916577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:19.917043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:19.917503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] PATH_INFO: `/v3/system/users/e2dc4cf3197f481daf10790ae1a9f50d/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:19.919413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=e2dc4cf3197f481daf10790ae1a9f50d, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:19.921271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:19.943423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 158/311] 10.222.0.42 () {58 vars in 1334 bytes} [Tue Jul 26 02:57:19 2022] PUT /identity/v3/system/users/e2dc4cf3197f481daf10790ae1a9f50d/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 02:57:19.971075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 154/312] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:19.986334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:20.108510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fbd069-7a67-4e8b-b052-5b6466ddb9ed None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:20.133250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:20.147585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:20.148244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:20.148515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:20.148794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:20.160058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:20.165501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:20.166583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:20.167732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 159/313] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:20.172804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a918c-bad8-46ce-a627-add4af1eaef4 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:20.277798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4a918c-bad8-46ce-a627-add4af1eaef4 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:20.297569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:20.318755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:20.319423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:20.319701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:20.319974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:20.320721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:20.321512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:20.322450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:20.332091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 155/314] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:20.337772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:20.438488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77124f62-062f-45ce-ba6c-04701af1db85 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:20.472471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:20.473133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:20.473410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:20.473685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] PATH_INFO: `/v3/users/036b8d9607994e19a57f546ebdfc5e9d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:20.474849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] RBAC: Authorizing `identity:get_user(user_id=036b8d9607994e19a57f546ebdfc5e9d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:20.476353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:20.477970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 160/315] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:20 2022] GET /identity/v3/users/036b8d9607994e19a57f546ebdfc5e9d => generated 297 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:20.483821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:20.601342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63854294-1d8d-4b70-96dd-e70be31b12e4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:20.635599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:20.636527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:20.636973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:20.637409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:20.686019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:20.709601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:20.710998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:20.812373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 156/316] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:20.817972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:20.870628 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:20.921564 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab8bcbe-9782-4303-8b3d-8db9916f3ba1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:20.956694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edccbd47-f526-4a67-b4f6-ade463be9243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:20.957641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edccbd47-f526-4a67-b4f6-ade463be9243 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:20.958090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edccbd47-f526-4a67-b4f6-ade463be9243 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:20.958529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edccbd47-f526-4a67-b4f6-ade463be9243 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:20.959971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edccbd47-f526-4a67-b4f6-ade463be9243 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:20.961599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edccbd47-f526-4a67-b4f6-ade463be9243 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:20.963794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 161/317] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:20 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:20.971965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:21.053718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:21.076541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb2a39f-9c2c-4896-b239-b4aa90bc939e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:21.111383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82666599-caea-4d61-8baf-a72904543705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:21.112014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82666599-caea-4d61-8baf-a72904543705 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:21.112291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82666599-caea-4d61-8baf-a72904543705 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:21.112562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82666599-caea-4d61-8baf-a72904543705 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:21.113364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82666599-caea-4d61-8baf-a72904543705 None admin] RBAC: Adding query filter params (role.id=7b22f4d466c74886bb0103bf3f7b59d1, scope.project.id=1f9b890871df4d1f96f60a844ee6b347, user.id=036b8d9607994e19a57f546ebdfc5e9d) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:21.113670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82666599-caea-4d61-8baf-a72904543705 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:21.115140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82666599-caea-4d61-8baf-a72904543705 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:21.122229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 157/318] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:20 2022] GET /identity/v3/role_assignments?user.id=036b8d9607994e19a57f546ebdfc5e9d&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=7b22f4d466c74886bb0103bf3f7b59d1 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:21.652982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80ac105-e194-43a6-a71f-855a125edbbe None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:21.653404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80ac105-e194-43a6-a71f-855a125edbbe None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:21.653611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80ac105-e194-43a6-a71f-855a125edbbe None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:21.654920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 162/319] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:21.660627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9866c43-ecbf-485e-ae75-64e854522395 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:21.660919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9866c43-ecbf-485e-ae75-64e854522395 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:21.661185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9866c43-ecbf-485e-ae75-64e854522395 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:21.680939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a9866c43-ecbf-485e-ae75-64e854522395 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:21.682142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a9866c43-ecbf-485e-ae75-64e854522395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:21.753658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 158/320] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:21.775401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80ac105-e194-43a6-a71f-855a125edbbe None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:21.883483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80ac105-e194-43a6-a71f-855a125edbbe None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:21.891422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3eb47e-0d27-4f6d-adcd-5a36e3a0524b None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:21.891692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3eb47e-0d27-4f6d-adcd-5a36e3a0524b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:21.891979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3eb47e-0d27-4f6d-adcd-5a36e3a0524b None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:21.893488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 159/321] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:21.898464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41e1885-0758-4f87-853b-21f6391a1869 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:21.898731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41e1885-0758-4f87-853b-21f6391a1869 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:21.899046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41e1885-0758-4f87-853b-21f6391a1869 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:21.901501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:21.920956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e41e1885-0758-4f87-853b-21f6391a1869 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:21.921144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:21.921708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:21.922231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:21.922231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:21.922354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e41e1885-0758-4f87-853b-21f6391a1869 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:21.923377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:21.924822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:21.926421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 163/322] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:21.931988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.016970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 160/323] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:22.038138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41e1885-0758-4f87-853b-21f6391a1869 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.040413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:22.053750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9e5d6a-b526-4dc3-94a4-ca271ccda0ab None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.086109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b24c04-736d-4e32-bb62-4c9b4d1fa0d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.086781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b24c04-736d-4e32-bb62-4c9b4d1fa0d8 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.087089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b24c04-736d-4e32-bb62-4c9b4d1fa0d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.087347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b24c04-736d-4e32-bb62-4c9b4d1fa0d8 None admin] PATH_INFO: `/v3/groups` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.088020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b24c04-736d-4e32-bb62-4c9b4d1fa0d8 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:22.089474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b24c04-736d-4e32-bb62-4c9b4d1fa0d8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:22.106823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 164/324] 10.222.0.42 () {60 vars in 1134 bytes} [Tue Jul 26 02:57:21 2022] POST /identity/v3/groups => generated 226 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:57:22.108123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8175ed66-48f3-4815-bfe4-01a1f66bca98 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.108406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8175ed66-48f3-4815-bfe4-01a1f66bca98 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.108677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8175ed66-48f3-4815-bfe4-01a1f66bca98 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.110091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 165/325] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:22.115769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395cd1d3-4d56-4610-a6c0-9b8e51ccaf91 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.116087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395cd1d3-4d56-4610-a6c0-9b8e51ccaf91 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.116333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395cd1d3-4d56-4610-a6c0-9b8e51ccaf91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.136149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-395cd1d3-4d56-4610-a6c0-9b8e51ccaf91 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:22.137865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-395cd1d3-4d56-4610-a6c0-9b8e51ccaf91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:22.149074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41e1885-0758-4f87-853b-21f6391a1869 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.188028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.189069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.189550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.190092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.190335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:22.206075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:22.213097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:22.214048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:22.214841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:22.218565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 166/326] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:22.220509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 161/327] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:22.224073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395cd1d3-4d56-4610-a6c0-9b8e51ccaf91 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.242087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.321719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:22.353515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395cd1d3-4d56-4610-a6c0-9b8e51ccaf91 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.378937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bfd6f2-2556-466f-ad7b-3afe3c7dbcb8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.398579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.399265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.399646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.399929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.400671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] RBAC: Adding query filter params (name=reader) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:22.401127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:22.402568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:22.414699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 167/328] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:22.421394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.426767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.427684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.428139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.428552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.441464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:22.446345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:22.447497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:22.448279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 162/329] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:22.456204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.479446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:22.574097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df72a3f-6a71-48ab-a88d-de8c35b132bf None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.597434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f991ab-51c4-496d-ba12-4b949db81574 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.618052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.618709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.619530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.619808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] PATH_INFO: `/v3/users/d78e6e040620489b8456d6d33062feae` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.621272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] RBAC: Authorizing `identity:get_user(user_id=d78e6e040620489b8456d6d33062feae)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:22.622473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:22.623902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:22.626224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 168/330] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:22 2022] GET /identity/v3/users/d78e6e040620489b8456d6d33062feae => generated 293 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:22.636373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.644043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.644683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.644960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.645229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.646021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:22.646327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:22.647777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:22.659090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 163/331] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:22.664472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.777115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284d3456-ceb4-42b5-9157-45776fe9022c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.784968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:22.792061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2e58a6-794b-4c13-b9b9-a5d8e5bebafb None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:22.813742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.814413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.814692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.814983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] PATH_INFO: `/v3/system/users/d78e6e040620489b8456d6d33062feae/roles/7b22f4d466c74886bb0103bf3f7b59d1` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.816931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=d78e6e040620489b8456d6d33062feae, role_id=7b22f4d466c74886bb0103bf3f7b59d1)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:22.818565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:22.829169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 169/332] 10.222.0.42 () {58 vars in 1334 bytes} [Tue Jul 26 02:57:22 2022] PUT /identity/v3/system/users/d78e6e040620489b8456d6d33062feae/roles/7b22f4d466c74886bb0103bf3f7b59d1 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:22.831884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eed286d-e785-40af-81d7-5621e8a08dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:22.832879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed286d-e785-40af-81d7-5621e8a08dde None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:22.833399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed286d-e785-40af-81d7-5621e8a08dde None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:22.833881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed286d-e785-40af-81d7-5621e8a08dde None admin] PATH_INFO: `/v3/users/e2dc4cf3197f481daf10790ae1a9f50d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:22.835582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eed286d-e785-40af-81d7-5621e8a08dde None admin] RBAC: Authorizing `identity:get_user(user_id=e2dc4cf3197f481daf10790ae1a9f50d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:22.837451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eed286d-e785-40af-81d7-5621e8a08dde None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:22.843120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 164/333] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:22 2022] GET /identity/v3/users/e2dc4cf3197f481daf10790ae1a9f50d => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:22.847464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:22.964601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31636726-e85e-485d-b483-6cfd08f0d54b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:23.006151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9010ec49-3567-43a2-9f1e-d8ca02620d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:23.007159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9010ec49-3567-43a2-9f1e-d8ca02620d04 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:23.007631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9010ec49-3567-43a2-9f1e-d8ca02620d04 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:23.008067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9010ec49-3567-43a2-9f1e-d8ca02620d04 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:23.009069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9010ec49-3567-43a2-9f1e-d8ca02620d04 None admin] RBAC: Adding query filter params (role.id=d0d63bdb855c485cabc4d6711c761e39, user.id=e2dc4cf3197f481daf10790ae1a9f50d, scope.system=all) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:23.009657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9010ec49-3567-43a2-9f1e-d8ca02620d04 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:23.011424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9010ec49-3567-43a2-9f1e-d8ca02620d04 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:23.025617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 170/334] 10.222.0.42 () {56 vars in 1308 bytes} [Tue Jul 26 02:57:22 2022] GET /identity/v3/role_assignments?user.id=e2dc4cf3197f481daf10790ae1a9f50d&scope.system=all&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 510 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:24.392190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abd35db-684b-4d86-ba6b-017f2e0b084d None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:24.392708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abd35db-684b-4d86-ba6b-017f2e0b084d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:24.392708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abd35db-684b-4d86-ba6b-017f2e0b084d None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:24.394401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 165/335] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:24.399463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3e4ece-0c3b-48a4-80fa-216c56e421c2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:24.399871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3e4ece-0c3b-48a4-80fa-216c56e421c2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:24.400040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3e4ece-0c3b-48a4-80fa-216c56e421c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:24.423947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3f3e4ece-0c3b-48a4-80fa-216c56e421c2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:24.425138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3f3e4ece-0c3b-48a4-80fa-216c56e421c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:24.456330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:24.520565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 171/336] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:24.560182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abd35db-684b-4d86-ba6b-017f2e0b084d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:24.646450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:24.724645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abd35db-684b-4d86-ba6b-017f2e0b084d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:24.772611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc15bd2-bb58-4bbe-915e-f85640f499ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:24.773271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc15bd2-bb58-4bbe-915e-f85640f499ec None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:24.773540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc15bd2-bb58-4bbe-915e-f85640f499ec None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:24.773812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc15bd2-bb58-4bbe-915e-f85640f499ec None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:24.788130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc15bd2-bb58-4bbe-915e-f85640f499ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-0cc15bd2-bb58-4bbe-915e-f85640f499ec None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:24.794176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:24.795090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:24.795888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 166/337] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:24.801528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3e4ece-0c3b-48a4-80fa-216c56e421c2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:24.949310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3e4ece-0c3b-48a4-80fa-216c56e421c2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:25.000629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:25.001658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.002130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.002593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.003554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:25.004071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:25.005675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:25.029166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fbfb58-e6b0-4373-9f93-7c57a042326b None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.029602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 172/338] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:25.029987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fbfb58-e6b0-4373-9f93-7c57a042326b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.030472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fbfb58-e6b0-4373-9f93-7c57a042326b None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.032269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 167/339] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:25.038631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:25.043342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46772103-81b5-414d-a04f-1a7217d331fc None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.043909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46772103-81b5-414d-a04f-1a7217d331fc None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.044389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46772103-81b5-414d-a04f-1a7217d331fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.081611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:25.087796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-46772103-81b5-414d-a04f-1a7217d331fc None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:25.089123 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-46772103-81b5-414d-a04f-1a7217d331fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:25.261151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 168/340] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:25.268081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acc6136-666a-4739-836b-c9d87e74ad05 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:25.304632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46772103-81b5-414d-a04f-1a7217d331fc None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:25.338912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:25.339577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.339859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.340134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.397833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:25.495735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:25.497352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:25.520594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46772103-81b5-414d-a04f-1a7217d331fc None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:25.542586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:25.589257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d90cc6-a31b-40c5-9799-35f19e7ba9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:25.589928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d90cc6-a31b-40c5-9799-35f19e7ba9b5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.590204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d90cc6-a31b-40c5-9799-35f19e7ba9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.590479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d90cc6-a31b-40c5-9799-35f19e7ba9b5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.610671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d90cc6-a31b-40c5-9799-35f19e7ba9b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-28d90cc6-a31b-40c5-9799-35f19e7ba9b5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:25.616458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:25.617331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:25.618148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 169/341] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:25.619536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75761d79-6c55-4843-959a-d713235612b9 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.620768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75761d79-6c55-4843-959a-d713235612b9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.620768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75761d79-6c55-4843-959a-d713235612b9 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.622166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 170/342] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:25.627928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b04ae-e96d-4251-8998-564896305a3d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.628211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b04ae-e96d-4251-8998-564896305a3d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.628486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4b04ae-e96d-4251-8998-564896305a3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.650780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-df4b04ae-e96d-4251-8998-564896305a3d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:25.652027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-df4b04ae-e96d-4251-8998-564896305a3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:25.723399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 173/343] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 685 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:25.724456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:25.736687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 171/344] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:25.738028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456ba27-d1f0-4c1b-82a9-44e23899bb63 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.738312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456ba27-d1f0-4c1b-82a9-44e23899bb63 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.738590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5456ba27-d1f0-4c1b-82a9-44e23899bb63 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.740147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 172/345] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:25.740689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5456ba27-d1f0-4c1b-82a9-44e23899bb63 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:25.779855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:25.863181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c49982-dc57-4fa9-9190-566a82145935 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:25.873805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5456ba27-d1f0-4c1b-82a9-44e23899bb63 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:25.902533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:25.926845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:25.927565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.927847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.928217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.929171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:25.929476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:25.930876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5f8bc6-3c70-47d7-b9f7-4eac1c93221a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:25.931106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:25.931571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f8bc6-3c70-47d7-b9f7-4eac1c93221a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.931854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f8bc6-3c70-47d7-b9f7-4eac1c93221a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.932133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f8bc6-3c70-47d7-b9f7-4eac1c93221a None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.935407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5f8bc6-3c70-47d7-b9f7-4eac1c93221a None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:25.937571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5f8bc6-3c70-47d7-b9f7-4eac1c93221a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:25.940206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 173/346] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:25 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:25.941870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab44e56-935d-40ee-adb6-a8e7f113b41d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:25.942158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab44e56-935d-40ee-adb6-a8e7f113b41d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:25.942484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab44e56-935d-40ee-adb6-a8e7f113b41d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:25.943016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 174/347] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:25.943874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:25.976307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-bab44e56-935d-40ee-adb6-a8e7f113b41d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:25.977544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bab44e56-935d-40ee-adb6-a8e7f113b41d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:26.099273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 174/348] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:26.100080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab44e56-935d-40ee-adb6-a8e7f113b41d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:26.108267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a755680-5b6d-4b10-b631-ede762e5e507 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.153853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.154677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.155081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.155442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.156964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:26.168968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:26.175020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:26.176117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:26.176928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 175/349] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:26.177592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:26.233932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:26.266974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab44e56-935d-40ee-adb6-a8e7f113b41d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.317996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.318428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f7ead5-ab0b-4e6f-a547-13f1cc7f5073 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.318632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.318911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.319962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.362375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.364077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.364077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.364077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.377734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:26.380262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:26.382237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 176/350] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:26.382895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:26.424505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:26.426156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:26.463588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:26.515680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c923070f-8c4a-4362-a88d-b9cff2190be7 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.525333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:26.557211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 175/351] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:26.558525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb7f7c9-93d0-4f00-8df9-b37416c84661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.559225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7f7c9-93d0-4f00-8df9-b37416c84661 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.559572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7f7c9-93d0-4f00-8df9-b37416c84661 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.559851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb7f7c9-93d0-4f00-8df9-b37416c84661 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.560851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:26.573092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb7f7c9-93d0-4f00-8df9-b37416c84661 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-5cb7f7c9-93d0-4f00-8df9-b37416c84661 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:26.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:26.581080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:26.581875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 177/352] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:26.584618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1b24c9-8f89-449d-b21f-6d1a31e1f11e None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.584618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1b24c9-8f89-449d-b21f-6d1a31e1f11e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.584618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1b24c9-8f89-449d-b21f-6d1a31e1f11e None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.587576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 178/353] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:26.587576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1b24c9-8f89-449d-b21f-6d1a31e1f11e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:26.695494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983d7260-93c5-4ac0-a3a3-18adaeec8440 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.721455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1b24c9-8f89-449d-b21f-6d1a31e1f11e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.734781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:26.741194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.743385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.743714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.744306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.745053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:26.745365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:26.747310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:26.762630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 176/354] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:26.763804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:26.767994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.768647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.768932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.769202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.770010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1e52b22e640c409e9307765a7cc712c4) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:26.770363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:26.771830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:26.778510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 179/355] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 260 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:26.779246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:26.821067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:26.910876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fedb370-a920-426e-8673-69869e031e63 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.919490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5db0319-cc96-41f5-87ea-c438f7de7a47 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:26.961574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9e3889-f99c-4d89-9f8f-3b3ebfe660e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.962127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e3889-f99c-4d89-9f8f-3b3ebfe660e5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.962411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e3889-f99c-4d89-9f8f-3b3ebfe660e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.962923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e3889-f99c-4d89-9f8f-3b3ebfe660e5 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.963452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9e3889-f99c-4d89-9f8f-3b3ebfe660e5 None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:26.963761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9e3889-f99c-4d89-9f8f-3b3ebfe660e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:26.965152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9e3889-f99c-4d89-9f8f-3b3ebfe660e5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:26.975297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:26.976140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 180/356] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:26.976531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.976999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.977415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee49fe20-8941-49bb-9c17-efc34486add7 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:26.977703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee49fe20-8941-49bb-9c17-efc34486add7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:26.977911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.978015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee49fe20-8941-49bb-9c17-efc34486add7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:26.983728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:26.985574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:26.987899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 177/357] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:26.988816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.006544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ee49fe20-8941-49bb-9c17-efc34486add7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:27.006544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ee49fe20-8941-49bb-9c17-efc34486add7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:27.072350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:27.101710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 181/358] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:27.103010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee49fe20-8941-49bb-9c17-efc34486add7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.123962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afc3511-eb51-4174-8a77-05a08264a469 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:27.134291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:27.163844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:27.164184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.164480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.164738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] PATH_INFO: `/v3/users/d78e6e040620489b8456d6d33062feae` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.165946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] RBAC: Authorizing `identity:get_user(user_id=d78e6e040620489b8456d6d33062feae)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:27.167462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:27.169141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 178/359] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:57:26 2022] GET /identity/v3/users/d78e6e040620489b8456d6d33062feae => generated 293 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:27.170063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.231793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee49fe20-8941-49bb-9c17-efc34486add7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:27.276466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:27.277114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.277388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.277688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.298896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53c7ac1-4352-4766-a18b-4c694b3f2e71 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:27.338696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:27.352528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e958de-03d1-4fdf-bfaf-f1c92167f33d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:27.353325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e958de-03d1-4fdf-bfaf-f1c92167f33d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.353781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e958de-03d1-4fdf-bfaf-f1c92167f33d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.354063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e958de-03d1-4fdf-bfaf-f1c92167f33d None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.355641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e958de-03d1-4fdf-bfaf-f1c92167f33d None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:27.357477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e958de-03d1-4fdf-bfaf-f1c92167f33d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:27.359669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 179/360] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:27 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:27.361458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a2cfb-48eb-446e-8120-47b22d9e6345 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.361751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a2cfb-48eb-446e-8120-47b22d9e6345 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.362026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a2cfb-48eb-446e-8120-47b22d9e6345 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.363268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:27.363491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 180/361] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:27.364196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602a2cfb-48eb-446e-8120-47b22d9e6345 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.364876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:27.393402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:27.488680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 182/362] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:27.489484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.509079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602a2cfb-48eb-446e-8120-47b22d9e6345 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:27.561381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:27.562065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.562350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.562617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.578447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:27.585933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:27.586843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 181/363] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:27.588101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.611399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbe396a-e8bd-4a69-a542-df7b8544d6d5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:27.643317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:27.660426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea17d6d2-6614-4be1-ae6e-6e8ecf746745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:27.661071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea17d6d2-6614-4be1-ae6e-6e8ecf746745 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.661353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea17d6d2-6614-4be1-ae6e-6e8ecf746745 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.661633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea17d6d2-6614-4be1-ae6e-6e8ecf746745 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.662481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea17d6d2-6614-4be1-ae6e-6e8ecf746745 None admin] RBAC: Adding query filter params (role.id=7b22f4d466c74886bb0103bf3f7b59d1, user.id=d78e6e040620489b8456d6d33062feae, scope.system=all) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:27.662789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea17d6d2-6614-4be1-ae6e-6e8ecf746745 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:27.664374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea17d6d2-6614-4be1-ae6e-6e8ecf746745 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:27.671331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:27.682181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 183/364] 10.222.0.42 () {56 vars in 1308 bytes} [Tue Jul 26 02:57:27 2022] GET /identity/v3/role_assignments?user.id=d78e6e040620489b8456d6d33062feae&scope.system=all&role.id=7b22f4d466c74886bb0103bf3f7b59d1 => generated 510 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:27.683524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244694a-11e1-445a-aa64-2cc2767eb0b1 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.683811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244694a-11e1-445a-aa64-2cc2767eb0b1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.684087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244694a-11e1-445a-aa64-2cc2767eb0b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.713047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4244694a-11e1-445a-aa64-2cc2767eb0b1 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:27.714341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4244694a-11e1-445a-aa64-2cc2767eb0b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:27.743793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231532b8-739f-4334-bcb7-23056d642ad3 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:27.792868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:27.793562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:27.793916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:27.794193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:27.795056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1e52b22e640c409e9307765a7cc712c4) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:27.795364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:27.796859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:27.800628 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 184/365] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:27.801879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4244694a-11e1-445a-aa64-2cc2767eb0b1 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.808995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 182/366] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:27 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=629444c0e927466f8da9275a0841de95 => generated 260 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:27.809947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:27.992788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8531adde-e51f-42d4-bd02-c4d6696a6bd3 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.009425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4244694a-11e1-445a-aa64-2cc2767eb0b1 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.025167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:28.027912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:28.047472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.048254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.048644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.048924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.049667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:28.050120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:28.051543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:28.058827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.059794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.060165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.060533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.061610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 183/367] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:28.062064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:28.062479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.063920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:28.066254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 185/368] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:27 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:28.071942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.190484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4dfc92-ca66-4895-bdd5-5622edac540c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.204637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780bc2b-a516-4a77-9fd2-e599c2e60ffa None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.228863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.229519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.229798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.230072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.251232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:28.252089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.255730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.256013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.256288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.257564 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:28.258564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:28.259648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:28.260525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:28.261111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:28.261111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 186/369] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:28.261552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.262722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 184/370] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:28.267993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.324232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:28.324769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:28.390420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70627ce7-ad9a-4b4e-a794-afb3182f1310 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.398971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2b8d7e-21db-431a-b0a8-b50610f41be1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.444387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.445315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.445592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.445688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.445856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.446261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.446544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.446822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.447679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1f9b890871df4d1f96f60a844ee6b347) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:28.447996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:28.449453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:28.458985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 185/371] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 260 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:28.458985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.574787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:28.579237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:28.632637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab24f87-4e86-45fa-a138-1f70a30ecedf None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.641341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:28.675710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.676355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.676635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.676909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.678238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:28.678786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:28.679136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:28.681734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:28.691277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 186/372] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:28.696591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.789106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 187/373] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 528 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:28.790172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.870797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72083e3e-44cf-4dba-bd51-76300b4b6553 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.905929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9637daf-a4ae-4416-a868-d23c7e03f816 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:28.943917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.944575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.944855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.945125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.946462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:28.948334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:28.950743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 188/374] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:28.958731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:28.971681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b29549-0f6f-430e-93b4-b45677cfcd1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:28.972617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b29549-0f6f-430e-93b4-b45677cfcd1d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:28.973089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b29549-0f6f-430e-93b4-b45677cfcd1d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:28.973529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b29549-0f6f-430e-93b4-b45677cfcd1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:28.992134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.070278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0f3cad-2755-4187-a1e1-e758bd565f9b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.074771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.120575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f35cab-0180-4959-ad5d-21d8969e15c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.122099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f35cab-0180-4959-ad5d-21d8969e15c4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.122448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f35cab-0180-4959-ad5d-21d8969e15c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.123534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f35cab-0180-4959-ad5d-21d8969e15c4 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.126173 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f35cab-0180-4959-ad5d-21d8969e15c4 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.128962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f35cab-0180-4959-ad5d-21d8969e15c4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.132158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 189/375] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:29.133418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7264bd-d08d-4e9a-8b11-5c3a42baf9ca None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.133703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7264bd-d08d-4e9a-8b11-5c3a42baf9ca None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.133980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7264bd-d08d-4e9a-8b11-5c3a42baf9ca None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.143326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 190/376] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:29 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:29.144705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7264bd-d08d-4e9a-8b11-5c3a42baf9ca None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:29.146745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b29549-0f6f-430e-93b4-b45677cfcd1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.148692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b29549-0f6f-430e-93b4-b45677cfcd1d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.285516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.314975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 187/377] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 617 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:57:29.314975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137f7038-561c-41d9-80b5-cca8606bd00a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.315262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137f7038-561c-41d9-80b5-cca8606bd00a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.315509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137f7038-561c-41d9-80b5-cca8606bd00a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.319426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7264bd-d08d-4e9a-8b11-5c3a42baf9ca None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.339351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-137f7038-561c-41d9-80b5-cca8606bd00a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:29.340893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-137f7038-561c-41d9-80b5-cca8606bd00a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:29.363711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.364344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.364622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.364899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.365703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1f9b890871df4d1f96f60a844ee6b347) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:29.366279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.367772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.374136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 191/378] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=629444c0e927466f8da9275a0841de95 => generated 260 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:29.374917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:29.413387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.423394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 188/379] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:29.447343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137f7038-561c-41d9-80b5-cca8606bd00a None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:29.498269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe497b0-27b0-4f29-888c-64b18081537b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.535855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.536529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.536809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.537085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] PATH_INFO: `/v3/projects/b8c4cb2e7662425d87dcbcca96df5974` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.538295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] RBAC: Authorizing `identity:get_project(project_id=b8c4cb2e7662425d87dcbcca96df5974)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.539936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.542048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 192/380] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/projects/b8c4cb2e7662425d87dcbcca96df5974 => generated 344 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:29.548855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:29.572166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.576479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137f7038-561c-41d9-80b5-cca8606bd00a None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.618752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.619714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.620175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.620582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.635664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:29.641718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:29.642773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:29.643723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:29.646240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 189/381] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:29.652651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:29.674345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5ad19-03e4-4bdf-a98a-3e6bd54ccbe8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.712005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.712685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.712858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.713077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] PATH_INFO: `/v3/groups/0a2c2ba4e3134efba8e7104340ae24f2` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.722972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.723169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] RBAC: Authorizing `identity:get_group(group_id=0a2c2ba4e3134efba8e7104340ae24f2)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.723169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.727202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 193/382] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/groups/0a2c2ba4e3134efba8e7104340ae24f2 => generated 229 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:29.731178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:29.768864 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e9865-8159-43b9-8007-088ab601037e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.805171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.806226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.806767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.807231 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.808125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] RBAC: Adding query filter params (name=member) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:29.808596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.810277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.816499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.822718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 190/383] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:29.828879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:29.841050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc25d07-2210-46ff-bb12-9cb39c7f9a58 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.881321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.881959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.882238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.882511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.883459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] RBAC: Adding query filter params (group.id=0a2c2ba4e3134efba8e7104340ae24f2, role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=b8c4cb2e7662425d87dcbcca96df5974) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:29.883772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.885323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.896112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 194/384] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/role_assignments?group.id=0a2c2ba4e3134efba8e7104340ae24f2&scope.project.id=b8c4cb2e7662425d87dcbcca96df5974&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 260 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:29.942579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca685ea9-74b2-424a-bc60-b45e033c46ad None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:29.966257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:29.982129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:29.982797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:29.983118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:29.983391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:29.984596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:29.986129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:29.988205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 191/385] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:29.995415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:30.133562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e92748-339f-48aa-9d23-03c7341fc7dd None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:30.173410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a251da7-ecb3-4e84-a9a1-7fcc3c53ac11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:30.174432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a251da7-ecb3-4e84-a9a1-7fcc3c53ac11 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:30.174907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a251da7-ecb3-4e84-a9a1-7fcc3c53ac11 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:30.175428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a251da7-ecb3-4e84-a9a1-7fcc3c53ac11 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:30.176905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a251da7-ecb3-4e84-a9a1-7fcc3c53ac11 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:30.178606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a251da7-ecb3-4e84-a9a1-7fcc3c53ac11 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:30.180974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 195/386] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:29 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:30.186585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:30.307294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da3930b-15d4-4e9d-9eb5-67919d43b4b1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:30.346133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7ca5f6-c5fa-44ce-ab05-b6fea9c47970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:30.346813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7ca5f6-c5fa-44ce-ab05-b6fea9c47970 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:30.347174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7ca5f6-c5fa-44ce-ab05-b6fea9c47970 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:30.347449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7ca5f6-c5fa-44ce-ab05-b6fea9c47970 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:30.349419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7ca5f6-c5fa-44ce-ab05-b6fea9c47970 None admin] RBAC: Authorizing `identity:create_grant(project_id=1e52b22e640c409e9307765a7cc712c4, group_id=4923d715652e4afca47a0bd1d8ba12ad, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:30.350862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7ca5f6-c5fa-44ce-ab05-b6fea9c47970 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:30.366284 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 192/387] 10.222.0.42 () {58 vars in 1442 bytes} [Tue Jul 26 02:57:30 2022] PUT /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 02:57:30.722142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ab9b3-ae9a-4fda-ae1a-978774acb436 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:30.723275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ab9b3-ae9a-4fda-ae1a-978774acb436 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:30.723763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ab9b3-ae9a-4fda-ae1a-978774acb436 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:30.725663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 196/388] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:30.730915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01df122-72c6-47b4-bccd-f934c1650d8f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:30.731507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01df122-72c6-47b4-bccd-f934c1650d8f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:30.731721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01df122-72c6-47b4-bccd-f934c1650d8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:30.760247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c01df122-72c6-47b4-bccd-f934c1650d8f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:30.761444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c01df122-72c6-47b4-bccd-f934c1650d8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:30.820554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:30.852077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 193/389] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:30.885783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2ab9b3-ae9a-4fda-ae1a-978774acb436 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:30.904000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:31.016687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2ab9b3-ae9a-4fda-ae1a-978774acb436 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:31.054497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1536245d-4bb5-4a7b-a858-041103d9e990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:31.055216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536245d-4bb5-4a7b-a858-041103d9e990 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.055498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536245d-4bb5-4a7b-a858-041103d9e990 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.055775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1536245d-4bb5-4a7b-a858-041103d9e990 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.068469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1536245d-4bb5-4a7b-a858-041103d9e990 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-1536245d-4bb5-4a7b-a858-041103d9e990 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:31.074084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:31.075167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:31.076026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 197/390] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:31.084101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01df122-72c6-47b4-bccd-f934c1650d8f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:31.158692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49461bf-5562-4694-9c05-a8dae69a0249 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.159062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49461bf-5562-4694-9c05-a8dae69a0249 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.159265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49461bf-5562-4694-9c05-a8dae69a0249 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.160883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 198/391] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:31.167366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234cb811-3025-4d8e-b205-bc05fedd199b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.167651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234cb811-3025-4d8e-b205-bc05fedd199b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.167930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234cb811-3025-4d8e-b205-bc05fedd199b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.199369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-234cb811-3025-4d8e-b205-bc05fedd199b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:31.200641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-234cb811-3025-4d8e-b205-bc05fedd199b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:31.215282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01df122-72c6-47b4-bccd-f934c1650d8f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:31.267035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:31.267642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.267929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.268204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.268938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:31.269245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:31.270636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:31.288742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 194/392] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:31.294607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:31.326102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 199/393] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:31.327340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:31.352437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:31.366791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234cb811-3025-4d8e-b205-bc05fedd199b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:31.460414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9997b938-2ad4-4563-8824-d91e5a37fbc1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:31.529503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:31.530447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.530919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.531423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.533416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:31.535057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:31.539249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234cb811-3025-4d8e-b205-bc05fedd199b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:31.541481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 195/394] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:31 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:31.544309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:31.613292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:31.614293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.614730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.615174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.635336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:31.642479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:31.643596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:31.644602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:31.652537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 200/395] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:31.655138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:31.739969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4038384-d325-4a7e-b3d7-21a6477ed12f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:31.756418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:31.761016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:31.803390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ad0b6-50bd-4f7f-9558-0f8c491eebdd None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:31.820548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:31.821199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.821477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.821824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.823285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:31.824670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:31.826737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 196/396] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:31 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:31.836064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:31.848756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:31.849694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:31.850179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:31.850619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:31.851594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:31.852083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:31.854617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:31.870132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 201/397] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:31.875879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:32.014004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c238940-2bb7-4f2a-a757-16d20804f9dc None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:32.053264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c572ae15-6275-41bd-8ef5-b24df8eb93c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:32.053264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572ae15-6275-41bd-8ef5-b24df8eb93c0 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.053264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572ae15-6275-41bd-8ef5-b24df8eb93c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.053264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c572ae15-6275-41bd-8ef5-b24df8eb93c0 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/629444c0e927466f8da9275a0841de95` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.056011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c572ae15-6275-41bd-8ef5-b24df8eb93c0 None admin] RBAC: Authorizing `identity:create_grant(project_id=1e52b22e640c409e9307765a7cc712c4, group_id=4923d715652e4afca47a0bd1d8ba12ad, role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:32.056182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2bfd6b-a61c-43fb-b73d-25be646b46c9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:32.057637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c572ae15-6275-41bd-8ef5-b24df8eb93c0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:32.072908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 197/398] 10.222.0.42 () {58 vars in 1442 bytes} [Tue Jul 26 02:57:31 2022] PUT /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/629444c0e927466f8da9275a0841de95 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:32.143234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c478fc1-fe56-4d7e-9bf6-8ee24142e100 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.143234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c478fc1-fe56-4d7e-9bf6-8ee24142e100 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.143234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c478fc1-fe56-4d7e-9bf6-8ee24142e100 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.145943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 198/399] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:32 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:32.148106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:32.149859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fae6df0-cfa1-431e-8c42-f06dcf4f1436 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.150239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fae6df0-cfa1-431e-8c42-f06dcf4f1436 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.150568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fae6df0-cfa1-431e-8c42-f06dcf4f1436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.188014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:32.189141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.189631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.190104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.191703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:32.193435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:32.196159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 202/400] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:31 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:32.202309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-7fae6df0-cfa1-431e-8c42-f06dcf4f1436 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:32.202515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:32.202604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7fae6df0-cfa1-431e-8c42-f06dcf4f1436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:32.271343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:32.320422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 199/401] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:32.345585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fae6df0-cfa1-431e-8c42-f06dcf4f1436 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:32.359494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab94c6be-8c78-498d-8b4e-51479ece2e03 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:32.406607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:32.407319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.407603 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.407981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.409180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:32.410739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:32.412600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 203/402] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:32 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:32.429693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:32.472822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fae6df0-cfa1-431e-8c42-f06dcf4f1436 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:32.519319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebfc932-802e-4b23-954a-d48a804b13ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:32.520186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebfc932-802e-4b23-954a-d48a804b13ed None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.520472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebfc932-802e-4b23-954a-d48a804b13ed None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.521176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebfc932-802e-4b23-954a-d48a804b13ed None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.538215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebfc932-802e-4b23-954a-d48a804b13ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:32.538567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-4ebfc932-802e-4b23-954a-d48a804b13ed None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:32.544607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:32.545509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:32.546281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:32.548144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 200/403] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:32.548144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e2273-5b58-47bc-8614-6934aba338c2 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.548144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e2273-5b58-47bc-8614-6934aba338c2 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.548620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2e2273-5b58-47bc-8614-6934aba338c2 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.550503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 201/404] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:32.551628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2e2273-5b58-47bc-8614-6934aba338c2 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:32.587024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:32.589866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e0dc76-ee97-4331-ac51-70e339092f35 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:32.628334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e2406-f9bf-4a3a-8b86-e8f368b44180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:32.629129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e2406-f9bf-4a3a-8b86-e8f368b44180 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.629475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e2406-f9bf-4a3a-8b86-e8f368b44180 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.629895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e2406-f9bf-4a3a-8b86-e8f368b44180 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.632167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e2406-f9bf-4a3a-8b86-e8f368b44180 None admin] RBAC: Authorizing `identity:create_grant(project_id=1f9b890871df4d1f96f60a844ee6b347, group_id=4923d715652e4afca47a0bd1d8ba12ad, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:32.633784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e2406-f9bf-4a3a-8b86-e8f368b44180 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:32.659066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 204/405] 10.222.0.42 () {58 vars in 1442 bytes} [Tue Jul 26 02:57:32 2022] PUT /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:32.660797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66df65-eb34-49e6-8fd4-c4764dce01ef None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.661095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66df65-eb34-49e6-8fd4-c4764dce01ef None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.661367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66df65-eb34-49e6-8fd4-c4764dce01ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.668735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2e2273-5b58-47bc-8614-6934aba338c2 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:32.695572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-cc66df65-eb34-49e6-8fd4-c4764dce01ef None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:32.696850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cc66df65-eb34-49e6-8fd4-c4764dce01ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:32.713421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:32.714073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.714349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.714617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.715425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:32.715778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:32.717188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:32.728301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 202/406] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:32.735772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:32.849785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 205/407] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:32.872919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66df65-eb34-49e6-8fd4-c4764dce01ef None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:32.899129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:32.920482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1423292-751d-4753-af89-9ef70ae79675 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:32.963396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:32.964058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:32.964340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:32.964620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:32.965876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:32.967529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:32.969591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 203/408] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:32 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:32.971514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:32.976869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.046821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66df65-eb34-49e6-8fd4-c4764dce01ef None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.083392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcb0fcd-887c-4568-a677-345d14c1ff61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.084182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcb0fcd-887c-4568-a677-345d14c1ff61 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.084542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcb0fcd-887c-4568-a677-345d14c1ff61 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.084880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcb0fcd-887c-4568-a677-345d14c1ff61 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.086349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dfbc24-58e6-4064-8f11-a6f9c267e06f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.096448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcb0fcd-887c-4568-a677-345d14c1ff61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-1dcb0fcd-887c-4568-a677-345d14c1ff61 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:33.102127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:33.103039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:33.104133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:33.108645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 206/409] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:33.109892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa9fa21-e306-406c-a081-47d70b82df27 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.110241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa9fa21-e306-406c-a081-47d70b82df27 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.110579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa9fa21-e306-406c-a081-47d70b82df27 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.112576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 207/410] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:33.113056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa9fa21-e306-406c-a081-47d70b82df27 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.127153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f020b6a-5f73-4187-b79f-c5ec39e298bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.128175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f020b6a-5f73-4187-b79f-c5ec39e298bd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.128638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f020b6a-5f73-4187-b79f-c5ec39e298bd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.129091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f020b6a-5f73-4187-b79f-c5ec39e298bd None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.130735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f020b6a-5f73-4187-b79f-c5ec39e298bd None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:33.132514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f020b6a-5f73-4187-b79f-c5ec39e298bd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:33.134892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 204/411] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:32 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:33.136351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17314d0-d1d1-4db0-80dc-2e7c4f9ae98d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.136819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17314d0-d1d1-4db0-80dc-2e7c4f9ae98d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.137259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17314d0-d1d1-4db0-80dc-2e7c4f9ae98d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.156705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d17314d0-d1d1-4db0-80dc-2e7c4f9ae98d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:33.158261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d17314d0-d1d1-4db0-80dc-2e7c4f9ae98d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:33.176961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:33.230607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa9fa21-e306-406c-a081-47d70b82df27 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.231048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 205/412] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:33.232119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17314d0-d1d1-4db0-80dc-2e7c4f9ae98d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.249585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:33.285327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.286291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.286761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.287242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.288152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:33.288596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:33.290144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:33.299482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 208/413] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:33.300527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.374004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17314d0-d1d1-4db0-80dc-2e7c4f9ae98d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.408630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3ff14a-d4b9-4473-91b6-98a3d74d42ce None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.417228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.417909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.418186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.418460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/629444c0e927466f8da9275a0841de95` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.420927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] RBAC: Authorizing `identity:create_grant(project_id=1f9b890871df4d1f96f60a844ee6b347, group_id=4923d715652e4afca47a0bd1d8ba12ad, role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:33.422431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:33.440311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 206/414] 10.222.0.42 () {58 vars in 1442 bytes} [Tue Jul 26 02:57:33 2022] PUT /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347/groups/4923d715652e4afca47a0bd1d8ba12ad/roles/629444c0e927466f8da9275a0841de95 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:33.441505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.447891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.448798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.449168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.449532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.463919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:33.464724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:33.471882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:33.473814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:33.475067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 209/415] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:33.492559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.562714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:33.654578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5640e580-b268-4448-8ae0-4dba2bf5e116 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.696814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14804db0-974c-49ad-bcdc-1e2fd5405528 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.708891 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.709979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.710256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.710528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] PATH_INFO: `/v3/groups/0a2c2ba4e3134efba8e7104340ae24f2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.712135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] RBAC: Authorizing `identity:get_group(group_id=0a2c2ba4e3134efba8e7104340ae24f2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:33.713874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:33.716834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 207/416] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:33 2022] GET /identity/v3/groups/0a2c2ba4e3134efba8e7104340ae24f2 => generated 229 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:33.723438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.748699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.749412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.749962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.750243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.751245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:33.751560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:33.753667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:33.765842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 210/417] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:33.772020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:33.863501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:33.884045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:33.893895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6385dbdb-0b87-4094-9453-13f0c48a90f8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.911134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868177ff-1db6-4bdb-b7d9-0baf31f9afe0 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:33.932529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.933172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.933455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.933735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:33.990774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:33.997069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:33.997355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:33.997635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] PATH_INFO: `/v3/projects/b8c4cb2e7662425d87dcbcca96df5974` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.003874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] RBAC: Authorizing `identity:get_project(project_id=b8c4cb2e7662425d87dcbcca96df5974)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:34.005551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:34.007421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 208/418] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:33 2022] GET /identity/v3/projects/b8c4cb2e7662425d87dcbcca96df5974 => generated 344 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:34.016136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:34.017731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:34.019717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:34.142215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:34.157453 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 211/419] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:34.163700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:34.212545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c797f-851c-4788-a22d-b3e8464db666 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:34.257531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:34.258220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.258499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.258777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] PATH_INFO: `/v3/projects/b8c4cb2e7662425d87dcbcca96df5974/groups/0a2c2ba4e3134efba8e7104340ae24f2/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.264263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] RBAC: Authorizing `identity:create_grant(project_id=b8c4cb2e7662425d87dcbcca96df5974, group_id=0a2c2ba4e3134efba8e7104340ae24f2, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:34.265735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:34.301560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 209/420] 10.222.0.42 () {58 vars in 1442 bytes} [Tue Jul 26 02:57:34 2022] PUT /identity/v3/projects/b8c4cb2e7662425d87dcbcca96df5974/groups/0a2c2ba4e3134efba8e7104340ae24f2/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:34.316708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90303c-8d54-43cb-b7cb-d76a084fecdf None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:34.355627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe38a36-eeff-4d63-9440-721d35c48469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:34.356769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe38a36-eeff-4d63-9440-721d35c48469 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.357242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe38a36-eeff-4d63-9440-721d35c48469 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.357730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe38a36-eeff-4d63-9440-721d35c48469 None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.359317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe38a36-eeff-4d63-9440-721d35c48469 None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:34.361033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe38a36-eeff-4d63-9440-721d35c48469 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:34.363593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 212/421] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:34 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:34.372364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:34.374344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c605c-4f56-49fc-882f-3ed23846436d None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.374864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c605c-4f56-49fc-882f-3ed23846436d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.375613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4c605c-4f56-49fc-882f-3ed23846436d None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.378159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 213/422] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:34.383731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986124a9-c765-435f-9556-770b1147ff7b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.384708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986124a9-c765-435f-9556-770b1147ff7b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.385235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986124a9-c765-435f-9556-770b1147ff7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.391476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:34.420344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-986124a9-c765-435f-9556-770b1147ff7b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:34.421506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-986124a9-c765-435f-9556-770b1147ff7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:34.535367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:34.553015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903ed6ba-24d5-4eda-a3da-cfa9d7c1ae3f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:34.553663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 214/423] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:34.576068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986124a9-c765-435f-9556-770b1147ff7b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:34.593223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:34.594241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.595144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.595661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.597024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:34.598720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:34.601953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 210/424] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:34 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:34.607473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:34.699411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:34.723735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16b05c-fcec-4767-8f35-6f864f13d725 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:34.761017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3978ba4a-94cf-423f-9836-87aae626a585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:34.762043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978ba4a-94cf-423f-9836-87aae626a585 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.763210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978ba4a-94cf-423f-9836-87aae626a585 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.763691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978ba4a-94cf-423f-9836-87aae626a585 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.764700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3978ba4a-94cf-423f-9836-87aae626a585 None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1e52b22e640c409e9307765a7cc712c4) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:34.765205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3978ba4a-94cf-423f-9836-87aae626a585 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:34.766886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3978ba4a-94cf-423f-9836-87aae626a585 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:34.774195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 211/425] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:34 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 611 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:34.819553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986124a9-c765-435f-9556-770b1147ff7b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:34.882632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0331b3-017d-4976-b579-68217b16bf72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:34.883347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0331b3-017d-4976-b579-68217b16bf72 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.883631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0331b3-017d-4976-b579-68217b16bf72 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.883908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0331b3-017d-4976-b579-68217b16bf72 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.907772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0331b3-017d-4976-b579-68217b16bf72 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:34.908835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59cc611-0dda-4d05-a49e-768b60d38b4c None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.909123 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59cc611-0dda-4d05-a49e-768b60d38b4c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.909397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59cc611-0dda-4d05-a49e-768b60d38b4c None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.911001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 212/426] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-ab0331b3-017d-4976-b579-68217b16bf72 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:34.912782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:34.913657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:34.914812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 215/427] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:34.920170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59cc611-0dda-4d05-a49e-768b60d38b4c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:34.923416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9027129c-3448-4efb-92c6-7421d2a1b3cb None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:34.923702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9027129c-3448-4efb-92c6-7421d2a1b3cb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:34.923984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9027129c-3448-4efb-92c6-7421d2a1b3cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:34.945956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9027129c-3448-4efb-92c6-7421d2a1b3cb None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:34.947223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9027129c-3448-4efb-92c6-7421d2a1b3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:35.007387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:35.036543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 216/428] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:35.062256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9027129c-3448-4efb-92c6-7421d2a1b3cb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.112584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59cc611-0dda-4d05-a49e-768b60d38b4c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:35.155022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:35.155932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:35.156346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:35.156826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:35.157821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:35.158271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:35.159897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:35.170055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 213/429] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:35.173936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9027129c-3448-4efb-92c6-7421d2a1b3cb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:35.176394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.180554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:35.210980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:35.211787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:35.212188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:35.212561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:35.225360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:35.232062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:35.233168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 26 02:57:35.233977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:35.235169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 217/430] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:35.240305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.287610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:35.299586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7457ed-467e-4755-8776-3a69d88c7336 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:35.352426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:35.353103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:35.353454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:35.353653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:35.362601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2c073e-f928-4f95-a73c-21bffd9f33a9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:35.406826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:35.408189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:35.408716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:35.409187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:35.410649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] RBAC: Adding query filter params (name=member) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:35.411223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:35.413331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:35.425866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 218/431] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:35.431783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.441490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:35.443796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:35.447469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:35.557307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf13a41-0309-4fcb-bdf6-48596f36baae None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:35.559305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:35.569143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 214/432] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:57:35.574882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.616326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:35.617349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:35.617850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:35.618311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:35.707028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:35.709076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:35.736058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7b7b6-5464-4413-855f-c5bd8a1c1c29 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:35.739274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:35.782245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:35.782918 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:35.783229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:35.783509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] PATH_INFO: `/v3/projects/1e52b22e640c409e9307765a7cc712c4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:35.784682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] RBAC: Authorizing `identity:get_project(project_id=1e52b22e640c409e9307765a7cc712c4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:35.786890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:35.789449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 215/433] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/projects/1e52b22e640c409e9307765a7cc712c4 => generated 298 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:35.795983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.825583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 219/434] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:35.834816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.835142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:35.907541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6dae81-8ed8-4925-a2b0-10da1c9600ec None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:35.944554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:35.945290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:35.945644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:35.945923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:35.947092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:35.948715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:35.955055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 216/435] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:35.956715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:35.962137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdcf274-5f9d-408b-a0fb-ad59daa25434 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.016058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:36.017391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.018554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.019374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.019824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.021493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:36.024720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:36.031430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 220/436] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:36.034207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:36.080664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc74313-b13a-47b9-b17a-e8ca925707da None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.117856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e021b4-94a1-448c-bee0-88abb9a7c419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.118492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e021b4-94a1-448c-bee0-88abb9a7c419 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.118779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e021b4-94a1-448c-bee0-88abb9a7c419 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.119095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e021b4-94a1-448c-bee0-88abb9a7c419 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.119911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e021b4-94a1-448c-bee0-88abb9a7c419 None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1e52b22e640c409e9307765a7cc712c4) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:36.120227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e021b4-94a1-448c-bee0-88abb9a7c419 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:36.121784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e021b4-94a1-448c-bee0-88abb9a7c419 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:36.127199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:36.128599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 217/437] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:35 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1e52b22e640c409e9307765a7cc712c4&role.id=629444c0e927466f8da9275a0841de95 => generated 611 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:36.129470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf7a166-58dc-4cf5-a5cc-5061ed84b960 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.129651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf7a166-58dc-4cf5-a5cc-5061ed84b960 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.130094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf7a166-58dc-4cf5-a5cc-5061ed84b960 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.135130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 218/438] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:36.137322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5aca11-2d0e-4583-9825-09473d0c5ebb None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.137608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5aca11-2d0e-4583-9825-09473d0c5ebb None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.137933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5aca11-2d0e-4583-9825-09473d0c5ebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.164717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-7b5aca11-2d0e-4583-9825-09473d0c5ebb None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:36.166257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7b5aca11-2d0e-4583-9825-09473d0c5ebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:36.170589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba754bc3-ee00-40f2-bf63-5b75de5640e4 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.214295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.214983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.215268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.215717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.216938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:36.218417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:36.220967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 221/439] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:36 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:36.229262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:36.264675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 219/440] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:36.287377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5aca11-2d0e-4583-9825-09473d0c5ebb None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:36.329911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:36.361120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9598bb-7aba-4ff2-85f9-9c52b2be1033 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.405984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.406617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.406901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.407214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.408035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=d0d63bdb855c485cabc4d6711c761e39, scope.project.id=1f9b890871df4d1f96f60a844ee6b347) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:36.408351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:36.411486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:36.420217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 222/441] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:36 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=d0d63bdb855c485cabc4d6711c761e39 => generated 611 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:36.425573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5aca11-2d0e-4583-9825-09473d0c5ebb None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.467763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc35a73-43ce-4955-95e5-40c7e0bba3f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.468836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc35a73-43ce-4955-95e5-40c7e0bba3f9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.469379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc35a73-43ce-4955-95e5-40c7e0bba3f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.470197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc35a73-43ce-4955-95e5-40c7e0bba3f9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.483571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc35a73-43ce-4955-95e5-40c7e0bba3f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9dc35a73-43ce-4955-95e5-40c7e0bba3f9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:36.489251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:36.490209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:36.491029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:36.492200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 220/442] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:36.498403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:36.501582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:36.617541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207bdafa-556f-403b-bcc1-217b7da5152c None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.617710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072d31df-c1d0-4cdb-b1f2-a5e0b751e534 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.617806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207bdafa-556f-403b-bcc1-217b7da5152c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.618048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207bdafa-556f-403b-bcc1-217b7da5152c None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.619687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 221/443] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:36.625040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a385665c-d8de-4340-aeec-b591274ccc3c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.625335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a385665c-d8de-4340-aeec-b591274ccc3c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.625600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a385665c-d8de-4340-aeec-b591274ccc3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.651743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a385665c-d8de-4340-aeec-b591274ccc3c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:36.653000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a385665c-d8de-4340-aeec-b591274ccc3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:36.663206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.663206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.663206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.663206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.663675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:36.663920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:36.665344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:36.674280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 223/444] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:36.680098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:36.739132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 222/445] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:36.756446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:36.762754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a385665c-d8de-4340-aeec-b591274ccc3c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:36.793190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:36.804419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ed4bbc-f4b6-4eda-b604-e636b695aea9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.844249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.845366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.845705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.846044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.898033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a385665c-d8de-4340-aeec-b591274ccc3c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:36.917956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:36.919496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:36.932507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:36.933261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:36.933647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:36.933947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:36.944870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:36.950815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:36.951769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:36.952584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 223/446] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:36.957810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:36.988642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:37.035318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 224/447] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:57:37.040741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:37.046584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:37.063808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424f594-02a5-4721-9f4a-a872644cd9d3 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.103983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.104607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.104874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.105154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.105866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:37.106264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.107695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.116816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 224/448] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:37.122269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:37.156121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb5165-26eb-4a2a-bd8c-02bf5fe58d5e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.191459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.192107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.192381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.192651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] PATH_INFO: `/v3/projects/1f9b890871df4d1f96f60a844ee6b347` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.193804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] RBAC: Authorizing `identity:get_project(project_id=1f9b890871df4d1f96f60a844ee6b347)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.195279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.196940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 225/449] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:37 2022] GET /identity/v3/projects/1f9b890871df4d1f96f60a844ee6b347 => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:37.202455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:37.223629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd5bb14-fd38-4a34-8be6-ccb36782362a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.235614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:37.259631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.260255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.260537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.260814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.276267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:37.305317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1351c94b-83cc-4b40-80af-6896166133d2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.329691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.331425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.338874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.339551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.339831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.340100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] PATH_INFO: `/v3/groups/4923d715652e4afca47a0bd1d8ba12ad` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.341399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] RBAC: Authorizing `identity:get_group(group_id=4923d715652e4afca47a0bd1d8ba12ad)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.342830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.344456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 226/450] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:37 2022] GET /identity/v3/groups/4923d715652e4afca47a0bd1d8ba12ad => generated 226 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:37.350619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:37.430930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 225/451] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 26 02:57:37.436649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:37.446514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:37.449471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e5e266-322e-4777-9620-0e3cebf65006 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.481595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:37.484125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.484852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.485207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.485514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.486315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] RBAC: Adding query filter params (group.id=4923d715652e4afca47a0bd1d8ba12ad, role.id=629444c0e927466f8da9275a0841de95, scope.project.id=1f9b890871df4d1f96f60a844ee6b347) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:37.486618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.488101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.494056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 227/452] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:37 2022] GET /identity/v3/role_assignments?group.id=4923d715652e4afca47a0bd1d8ba12ad&scope.project.id=1f9b890871df4d1f96f60a844ee6b347&role.id=629444c0e927466f8da9275a0841de95 => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:37.539385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26b0148-1e83-464e-972c-b57a6290a4df None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.579241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.579903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.580180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.580449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] PATH_INFO: `/v3/projects/b8c4cb2e7662425d87dcbcca96df5974` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.581762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] RBAC: Authorizing `identity:get_project(project_id=b8c4cb2e7662425d87dcbcca96df5974)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.583277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.585006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 226/453] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:57:37 2022] GET /identity/v3/projects/b8c4cb2e7662425d87dcbcca96df5974 => generated 344 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:37.591318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:37.706876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4202b2-b6c7-4535-a1aa-ffbb3a9a1f34 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.741688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e04c60-f99d-41ba-970b-f77e06edf2ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.742426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e04c60-f99d-41ba-970b-f77e06edf2ce None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.742708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e04c60-f99d-41ba-970b-f77e06edf2ce None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.743006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e04c60-f99d-41ba-970b-f77e06edf2ce None admin] PATH_INFO: `/v3/groups/0a2c2ba4e3134efba8e7104340ae24f2` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.744180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e04c60-f99d-41ba-970b-f77e06edf2ce None admin] RBAC: Authorizing `identity:get_group(group_id=0a2c2ba4e3134efba8e7104340ae24f2)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.745667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e04c60-f99d-41ba-970b-f77e06edf2ce None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.747393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 228/454] 10.222.0.42 () {56 vars in 1180 bytes} [Tue Jul 26 02:57:37 2022] GET /identity/v3/groups/0a2c2ba4e3134efba8e7104340ae24f2 => generated 229 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:37.753604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:37.855871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac0a870-87ce-4316-b965-87f698c13737 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:37.871962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:37.894780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7805dcf-7603-468f-b078-8c9ecd2962a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:37.895427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7805dcf-7603-468f-b078-8c9ecd2962a3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:37.897352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7805dcf-7603-468f-b078-8c9ecd2962a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:37.897648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7805dcf-7603-468f-b078-8c9ecd2962a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:37.898458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7805dcf-7603-468f-b078-8c9ecd2962a3 None admin] RBAC: Adding query filter params (group.id=0a2c2ba4e3134efba8e7104340ae24f2, role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=b8c4cb2e7662425d87dcbcca96df5974) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:37.898768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7805dcf-7603-468f-b078-8c9ecd2962a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:37.900860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7805dcf-7603-468f-b078-8c9ecd2962a3 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:37.910940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 227/455] 10.222.0.42 () {56 vars in 1376 bytes} [Tue Jul 26 02:57:37 2022] GET /identity/v3/role_assignments?group.id=0a2c2ba4e3134efba8e7104340ae24f2&scope.project.id=b8c4cb2e7662425d87dcbcca96df5974&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 611 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:40.321962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc9e8a-6d38-4818-a06e-79083cc6b422 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:40.322459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc9e8a-6d38-4818-a06e-79083cc6b422 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:40.322459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efc9e8a-6d38-4818-a06e-79083cc6b422 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:40.324160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 229/456] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:40.329534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcab044-4e48-4e72-a7bc-f613aaa051b5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:40.330024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcab044-4e48-4e72-a7bc-f613aaa051b5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:40.330100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcab044-4e48-4e72-a7bc-f613aaa051b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:40.354575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-8bcab044-4e48-4e72-a7bc-f613aaa051b5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:40.355825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8bcab044-4e48-4e72-a7bc-f613aaa051b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:40.449193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 228/457] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:40.494979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efc9e8a-6d38-4818-a06e-79083cc6b422 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:40.495612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8938566-79a2-498c-a1dc-77584260e241 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:40.495902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8938566-79a2-498c-a1dc-77584260e241 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:40.496182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8938566-79a2-498c-a1dc-77584260e241 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:40.497910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 229/458] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:40.514490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668511f6-377d-418a-a8cb-3d3ce0c62d24 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:40.514793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668511f6-377d-418a-a8cb-3d3ce0c62d24 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:40.515186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668511f6-377d-418a-a8cb-3d3ce0c62d24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:40.541118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-668511f6-377d-418a-a8cb-3d3ce0c62d24 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:40.543939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-668511f6-377d-418a-a8cb-3d3ce0c62d24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:40.557448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:40.675347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 230/459] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:40.694216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efc9e8a-6d38-4818-a06e-79083cc6b422 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:40.699282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668511f6-377d-418a-a8cb-3d3ce0c62d24 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:40.740490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:40.772711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:40.773383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:40.773661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:40.773932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:40.807591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668511f6-377d-418a-a8cb-3d3ce0c62d24 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:40.808058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:40.809623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:40.845854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:40.846774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:40.847639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:40.847639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:40.847639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:40.847639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:40.852009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 230/460] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:40.853243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:40.855011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7944b2-35ca-4bac-bbb4-01549c17cd8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:40.856084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7944b2-35ca-4bac-bbb4-01549c17cd8b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:40.856568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7944b2-35ca-4bac-bbb4-01549c17cd8b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:40.857025 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7944b2-35ca-4bac-bbb4-01549c17cd8b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:40.877637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7944b2-35ca-4bac-bbb4-01549c17cd8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:40.879474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7944b2-35ca-4bac-bbb4-01549c17cd8b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-2b7944b2-35ca-4bac-bbb4-01549c17cd8b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:40.900225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:40.901131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:40.901961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:40.901961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:40.901961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:40.901961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:40.904495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 231/461] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:40.905913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b8986a-c676-42e1-b905-b2291ce74cb5 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:40.906386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b8986a-c676-42e1-b905-b2291ce74cb5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:40.906857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b8986a-c676-42e1-b905-b2291ce74cb5 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:40.908550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 232/462] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:40.914263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b8986a-c676-42e1-b905-b2291ce74cb5 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:40.968881 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:40.990469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d17a2e-6fc4-4b7d-ac82-c7caa6eb6673 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.030295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7fd07a-3801-4279-a4b7-bee4801469dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.030774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:41.030925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fd07a-3801-4279-a4b7-bee4801469dd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.031231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fd07a-3801-4279-a4b7-bee4801469dd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.031509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fd07a-3801-4279-a4b7-bee4801469dd None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.032216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7fd07a-3801-4279-a4b7-bee4801469dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:41.032553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7fd07a-3801-4279-a4b7-bee4801469dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.034104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7fd07a-3801-4279-a4b7-bee4801469dd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.043491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b8986a-c676-42e1-b905-b2291ce74cb5 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.056626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 231/463] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:41.056626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d980fe0-8b52-4de7-99e8-4dfd3d5b0c17 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.056626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d980fe0-8b52-4de7-99e8-4dfd3d5b0c17 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.056626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d980fe0-8b52-4de7-99e8-4dfd3d5b0c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.073060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8d980fe0-8b52-4de7-99e8-4dfd3d5b0c17 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:41.074244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8d980fe0-8b52-4de7-99e8-4dfd3d5b0c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:41.084661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f36a50-e329-4f79-9bc3-ca2fd1f3b9f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.085339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f36a50-e329-4f79-9bc3-ca2fd1f3b9f0 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.085629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f36a50-e329-4f79-9bc3-ca2fd1f3b9f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.085902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f36a50-e329-4f79-9bc3-ca2fd1f3b9f0 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.086608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f36a50-e329-4f79-9bc3-ca2fd1f3b9f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:41.086915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f36a50-e329-4f79-9bc3-ca2fd1f3b9f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.088364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f36a50-e329-4f79-9bc3-ca2fd1f3b9f0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.107208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 233/464] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:41.107208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7573971-54d7-4f07-9151-a5751c893e08 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.107208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7573971-54d7-4f07-9151-a5751c893e08 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.107208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7573971-54d7-4f07-9151-a5751c893e08 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.107208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 234/465] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:41.107208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7573971-54d7-4f07-9151-a5751c893e08 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:41.159278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 232/466] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:41.168102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d980fe0-8b52-4de7-99e8-4dfd3d5b0c17 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:41.218134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7573971-54d7-4f07-9151-a5751c893e08 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.253078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d275f5e-0ca2-428c-a1a7-e21d36311d91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.253641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d275f5e-0ca2-428c-a1a7-e21d36311d91 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.253921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d275f5e-0ca2-428c-a1a7-e21d36311d91 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.254188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d275f5e-0ca2-428c-a1a7-e21d36311d91 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.254872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d275f5e-0ca2-428c-a1a7-e21d36311d91 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.256474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d275f5e-0ca2-428c-a1a7-e21d36311d91 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.256700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:41.279640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 235/467] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:41 2022] POST /identity/v3/users => generated 246 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:41.280889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99e8b93-c1f4-4576-a123-708b3d2d8057 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.281170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99e8b93-c1f4-4576-a123-708b3d2d8057 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.281441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99e8b93-c1f4-4576-a123-708b3d2d8057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.295035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:41.301723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c99e8b93-c1f4-4576-a123-708b3d2d8057 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:41.303380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c99e8b93-c1f4-4576-a123-708b3d2d8057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:41.304786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d980fe0-8b52-4de7-99e8-4dfd3d5b0c17 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.352095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.354467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.354920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.355372 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.356246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.358108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.389560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 233/468] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:41 2022] POST /identity/v3/users => generated 248 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:41.390624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:41.397010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 236/469] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:41.437042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99e8b93-c1f4-4576-a123-708b3d2d8057 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:41.522838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fa5ae1-b8ab-484a-94ea-f33d2e8f4b6a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.565709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:41.575173 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.576030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.576364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.576764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.592668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.595286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.596486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99e8b93-c1f4-4576-a123-708b3d2d8057 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:41.613070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:41.613999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:41.614890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:41.614890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:41.614890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:41.614890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:41.614890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 234/470] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:41.623063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:41.646900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.647618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.647897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.648176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.651356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:41.663956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.666077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:41.682481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:41.683446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:41.684337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:41.684337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:41.684337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:41.684337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:41.686709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 237/471] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:41.691786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:41.764838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4038db9c-11a9-41ce-9546-166b54a53a78 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.805856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89491cd-054b-4332-bd95-ca13915a0077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.807049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89491cd-054b-4332-bd95-ca13915a0077 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.807784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89491cd-054b-4332-bd95-ca13915a0077 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.808482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89491cd-054b-4332-bd95-ca13915a0077 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.809468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89491cd-054b-4332-bd95-ca13915a0077 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:41.810041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89491cd-054b-4332-bd95-ca13915a0077 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.812184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89491cd-054b-4332-bd95-ca13915a0077 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.817862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf0498b-c660-41ee-8921-4a80ee01dd90 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.827950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 235/472] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:41.829833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782096b4-f5fe-431e-8aac-d1132f4d932d None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.830292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782096b4-f5fe-431e-8aac-d1132f4d932d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.830684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782096b4-f5fe-431e-8aac-d1132f4d932d None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.832358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 236/473] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:41.833935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782096b4-f5fe-431e-8aac-d1132f4d932d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:41.860912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaab6861-9100-4340-b081-42fd64080dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:41.861977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaab6861-9100-4340-b081-42fd64080dfc None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.862161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaab6861-9100-4340-b081-42fd64080dfc None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.862526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaab6861-9100-4340-b081-42fd64080dfc None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.863552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaab6861-9100-4340-b081-42fd64080dfc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:41.863955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaab6861-9100-4340-b081-42fd64080dfc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:41.865540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaab6861-9100-4340-b081-42fd64080dfc None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:41.879848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 238/474] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:41.881561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02b562f-498f-4836-869c-3937d3b1140a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:41.882211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02b562f-498f-4836-869c-3937d3b1140a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:41.882681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02b562f-498f-4836-869c-3937d3b1140a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:41.889355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:41.904321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d02b562f-498f-4836-869c-3937d3b1140a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:41.906091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d02b562f-498f-4836-869c-3937d3b1140a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:41.963203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782096b4-f5fe-431e-8aac-d1132f4d932d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:41.973491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:42.007074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 239/475] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:42.008314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02b562f-498f-4836-869c-3937d3b1140a None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:42.031412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:42.032203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:42.032614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:42.032976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:42.033765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:42.035642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:42.063752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 237/476] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:41 2022] POST /identity/v3/users => generated 248 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:57:42.069943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:42.124279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02b562f-498f-4836-869c-3937d3b1140a None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:42.164923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:42.166174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:42.166631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:42.167751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:42.169135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:42.171145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:42.216604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 240/477] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:42 2022] POST /identity/v3/users => generated 249 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:42.264551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:42.269698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1edf72b-c7ab-4c0d-a193-a7d813078b64 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:42.312466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:42.313403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:42.313888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:42.314309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:42.315258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:42.316979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:42.326135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.sql.core [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 26 02:57:42.326135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 26 02:57:42.326135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [parameters: {'id': 'e905140dec0d4d9a9b246aca4e635ed9', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 26 02:57:42.326135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80428) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result = self._query(query) Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application conn.query(q) Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result.read() Jul 26 02:57:42.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return ref.to_dict() Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 26 02:57:42.346713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application next(self.gen) Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application yield resource Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application next(self.gen) Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.session.rollback() Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.force_reraise() Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise self.value Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application session.commit() Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 26 02:57:42.347904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._prepare_impl() Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.session.flush() Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._flush(objects) Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application compat.raise_( Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application flush_context.execute() Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rec.execute(self) Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 26 02:57:42.349077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 26 02:57:42.350277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result = self._query(query) Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application conn.query(q) Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result.read() Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 26 02:57:42.351415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [parameters: {'id': 'e905140dec0d4d9a9b246aca4e635ed9', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:42.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 26 02:57:42.354075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:42.357287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 238/478] 10.222.0.42 () {60 vars in 1131 bytes} [Tue Jul 26 02:57:42 2022] POST /identity/v3/roles => generated 144 bytes in 293 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:42.362906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:42.406621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:42.484584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6781f1-0d85-4e68-aef7-761e363c878c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:42.520055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bebb22c-76b9-4c98-8555-ac96e08f2251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:42.520711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bebb22c-76b9-4c98-8555-ac96e08f2251 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:42.520992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bebb22c-76b9-4c98-8555-ac96e08f2251 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:42.521265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bebb22c-76b9-4c98-8555-ac96e08f2251 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:42.533783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bebb22c-76b9-4c98-8555-ac96e08f2251 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9bebb22c-76b9-4c98-8555-ac96e08f2251 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:42.538713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:42.539661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:42.540444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 241/479] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:57:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:42.547416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:42.650783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfa80a4-7918-4d83-a73b-1645a48efbd6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:42.694144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef682386-1e98-426a-b0d2-a5b574416f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:42.695043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef682386-1e98-426a-b0d2-a5b574416f4a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:42.695460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef682386-1e98-426a-b0d2-a5b574416f4a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:42.695872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef682386-1e98-426a-b0d2-a5b574416f4a None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:42.696740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef682386-1e98-426a-b0d2-a5b574416f4a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:42.697212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef682386-1e98-426a-b0d2-a5b574416f4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:42.698730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef682386-1e98-426a-b0d2-a5b574416f4a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:42.710242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 239/480] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:43.725879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a52fbf5-437a-43b4-af83-3501c23c9214 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:43.726439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a52fbf5-437a-43b4-af83-3501c23c9214 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:43.726439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a52fbf5-437a-43b4-af83-3501c23c9214 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:43.728081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 242/481] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:43.733170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e3494-4dbb-4407-8a6d-eb1e8e079854 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:43.733992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e3494-4dbb-4407-8a6d-eb1e8e079854 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:43.734477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e3494-4dbb-4407-8a6d-eb1e8e079854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:43.767150 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f66e3494-4dbb-4407-8a6d-eb1e8e079854 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:43.768733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f66e3494-4dbb-4407-8a6d-eb1e8e079854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:43.795777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:43.858627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 240/482] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:43.904854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a52fbf5-437a-43b4-af83-3501c23c9214 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.019878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:44.033949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a52fbf5-437a-43b4-af83-3501c23c9214 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:44.082362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ce064a-b4c0-43e1-a086-a62621c88314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:44.083052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ce064a-b4c0-43e1-a086-a62621c88314 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.083333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ce064a-b4c0-43e1-a086-a62621c88314 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.083608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ce064a-b4c0-43e1-a086-a62621c88314 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.096316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82ce064a-b4c0-43e1-a086-a62621c88314 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-82ce064a-b4c0-43e1-a086-a62621c88314 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:44.101523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:44.102464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:44.103276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 243/483] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:44.110138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e3494-4dbb-4407-8a6d-eb1e8e079854 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.163236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fa76fc-0fe1-4c89-85fe-0f8c6dfbf818 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.163563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fa76fc-0fe1-4c89-85fe-0f8c6dfbf818 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.163793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fa76fc-0fe1-4c89-85fe-0f8c6dfbf818 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.165279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 244/484] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:44.172693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3503d7a0-0a20-49b8-a425-6cd4beefb801 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.173288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3503d7a0-0a20-49b8-a425-6cd4beefb801 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.173758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3503d7a0-0a20-49b8-a425-6cd4beefb801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.194866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3503d7a0-0a20-49b8-a425-6cd4beefb801 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:44.196710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3503d7a0-0a20-49b8-a425-6cd4beefb801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:44.300342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e3494-4dbb-4407-8a6d-eb1e8e079854 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:44.307650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 245/485] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:44.341338 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:44.342342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.342851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.343332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.344243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:44.345084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:44.346656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:44.349321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3503d7a0-0a20-49b8-a425-6cd4beefb801 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.368895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 241/486] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:44.375222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.412508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:44.505768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:44.545182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3503d7a0-0a20-49b8-a425-6cd4beefb801 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:44.545644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e01b28-5ce8-4619-85f7-348d5500426f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:44.581404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:44.582421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.582916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.583437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.596517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:44.598327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:44.613499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:44.614375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:44.615232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:44.615232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:44.615232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:44.615232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:44.617468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 242/487] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:44.622777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.655717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1894fab6-5610-46be-a9e4-d361b4502f94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:44.656400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894fab6-5610-46be-a9e4-d361b4502f94 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.656685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894fab6-5610-46be-a9e4-d361b4502f94 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.656962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1894fab6-5610-46be-a9e4-d361b4502f94 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.715811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1894fab6-5610-46be-a9e4-d361b4502f94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-1894fab6-5610-46be-a9e4-d361b4502f94 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:44.721589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:44.722505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:44.723307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 246/488] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 374 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:44.724516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221460d-ee08-45dc-94b8-0adc2f084a75 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.724625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221460d-ee08-45dc-94b8-0adc2f084a75 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.724846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221460d-ee08-45dc-94b8-0adc2f084a75 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.726345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 247/489] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:44.731352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:44.731834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5221460d-ee08-45dc-94b8-0adc2f084a75 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.747056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d9099-9e1b-4428-ba1e-37eb93f982ee None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:44.781085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74ec344-2aa1-4e5e-bf9b-d8c962fe5bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:44.781707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74ec344-2aa1-4e5e-bf9b-d8c962fe5bd4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.781989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74ec344-2aa1-4e5e-bf9b-d8c962fe5bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.782257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74ec344-2aa1-4e5e-bf9b-d8c962fe5bd4 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.782989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74ec344-2aa1-4e5e-bf9b-d8c962fe5bd4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:44.783299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74ec344-2aa1-4e5e-bf9b-d8c962fe5bd4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:44.784668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74ec344-2aa1-4e5e-bf9b-d8c962fe5bd4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:44.801882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 243/490] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:44.801882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e31b23-930a-4d0d-845b-ccccc016dbb2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.801882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e31b23-930a-4d0d-845b-ccccc016dbb2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.801882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e31b23-930a-4d0d-845b-ccccc016dbb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.818869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-42e31b23-930a-4d0d-845b-ccccc016dbb2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:44.820087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-42e31b23-930a-4d0d-845b-ccccc016dbb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:44.845415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5221460d-ee08-45dc-94b8-0adc2f084a75 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:44.878692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0835c42e-9e0f-4679-8cc3-ac484298050b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:44.879352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835c42e-9e0f-4679-8cc3-ac484298050b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.879635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835c42e-9e0f-4679-8cc3-ac484298050b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.879905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835c42e-9e0f-4679-8cc3-ac484298050b None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.880638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0835c42e-9e0f-4679-8cc3-ac484298050b None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:44.880987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0835c42e-9e0f-4679-8cc3-ac484298050b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:44.882300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0835c42e-9e0f-4679-8cc3-ac484298050b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:44.890442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 248/491] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:44.891647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe65ded-eee9-4922-a0bd-e2e78a8b3985 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:44.891925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe65ded-eee9-4922-a0bd-e2e78a8b3985 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:44.892197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe65ded-eee9-4922-a0bd-e2e78a8b3985 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:44.893531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 249/492] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:44.894058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe65ded-eee9-4922-a0bd-e2e78a8b3985 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.903675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:44.911079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 244/493] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:44.912869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e31b23-930a-4d0d-845b-ccccc016dbb2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:44.984387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.014643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe65ded-eee9-4922-a0bd-e2e78a8b3985 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.038379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e31b23-930a-4d0d-845b-ccccc016dbb2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.053679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2e00a3-d38c-41de-a870-98e20ba33829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.054354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2e00a3-d38c-41de-a870-98e20ba33829 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.054628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2e00a3-d38c-41de-a870-98e20ba33829 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.055190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2e00a3-d38c-41de-a870-98e20ba33829 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.064372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2e00a3-d38c-41de-a870-98e20ba33829 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.065825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2e00a3-d38c-41de-a870-98e20ba33829 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.075192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.075909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.076316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.076690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-ca2e00a3-d38c-41de-a870-98e20ba33829 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:45.080235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:45.081187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:45.082294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 250/494] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:57:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:45.083229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7c130-cdae-45e0-ae9b-0f1a267acea0 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.083229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7c130-cdae-45e0-ae9b-0f1a267acea0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.090112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb7c130-cdae-45e0-ae9b-0f1a267acea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.096676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.099102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.104888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0cb7c130-cdae-45e0-ae9b-0f1a267acea0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:45.106184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0cb7c130-cdae-45e0-ae9b-0f1a267acea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:45.114981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:45.115849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:45.121726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:45.121726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:45.121726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:45.121726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:45.121726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 245/495] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:45.121726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.178742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.198981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 251/496] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:45.198981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7c130-cdae-45e0-ae9b-0f1a267acea0 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.239156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a70424-83bc-44e3-9c87-25c3f58e8ac2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.250135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.274784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.275467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.275746 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.276430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.287758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:45.293175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:45.294106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:45.294893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 246/497] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:45.296072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.308550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb7c130-cdae-45e0-ae9b-0f1a267acea0 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.346053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.346733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.346978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.347249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.348055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:45.348358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.349805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.391329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 252/498] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:45.392198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.416299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af07057-dc66-4c58-89b9-e88bceea1bbb None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.456811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.457707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.457985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.458259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.459145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:45.459313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.459798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.461083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.474523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 247/499] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:45.479802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.507645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c582b7-64d3-4090-93b7-caae80a0039d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.511933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.541998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.542638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.542916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.543230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.553437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:45.558480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:45.559442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 253/500] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:45.560486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.587067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bea3bb2-fe01-467b-b597-3b8a64fa173c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.621659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.622195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.622473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.622728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.623470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:45.623774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.625204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.633561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 248/501] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:45.634386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.662015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db209da-91d2-4184-af2a-c8b8646fda93 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.676473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.698663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.699572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.699857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.700129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.731025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.740293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ced7da-3d31-4e76-965f-a0a102f020e1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.772790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e632537e-3e05-4fe5-a18a-dee40f56f288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.773520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e632537e-3e05-4fe5-a18a-dee40f56f288 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.773712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e632537e-3e05-4fe5-a18a-dee40f56f288 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.773938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.774371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e632537e-3e05-4fe5-a18a-dee40f56f288 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.775462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.781664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e632537e-3e05-4fe5-a18a-dee40f56f288 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.783158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e632537e-3e05-4fe5-a18a-dee40f56f288 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e632537e-3e05-4fe5-a18a-dee40f56f288 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:45.795686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:45.796689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:45.797644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 249/502] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/users/glance => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:45.798186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c14f246-dd31-477c-b3bd-5c9cb2508e05 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.798249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c14f246-dd31-477c-b3bd-5c9cb2508e05 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.799217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c14f246-dd31-477c-b3bd-5c9cb2508e05 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.799813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 250/503] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:45 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:45.800507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c14f246-dd31-477c-b3bd-5c9cb2508e05 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.882144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 254/504] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:45.882933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.901277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.903445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c14f246-dd31-477c-b3bd-5c9cb2508e05 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:45.937791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:45.938418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:45.938702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:45.938998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:45.939728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:45.940037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:45.941445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:45.949643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 251/505] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:45.950442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:45.966518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:45.992474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6d6429-6f06-4c86-af1c-c8abd22529b8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.029186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa70d37-d0fa-4fc0-b999-f037f1c48c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.029836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa70d37-d0fa-4fc0-b999-f037f1c48c21 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.030115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa70d37-d0fa-4fc0-b999-f037f1c48c21 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.030378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa70d37-d0fa-4fc0-b999-f037f1c48c21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.040840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa70d37-d0fa-4fc0-b999-f037f1c48c21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.042346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa70d37-d0fa-4fc0-b999-f037f1c48c21 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.060538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b860e21e-0aad-4f42-a278-697fda0dac0e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-6aa70d37-d0fa-4fc0-b999-f037f1c48c21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:46.061616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:46.062472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:46.063732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:46.063732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:46.063732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.063732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:46.063732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 255/506] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:46.064228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5717803e-8d1e-428a-b81b-a0512607bb9b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.064306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5717803e-8d1e-428a-b81b-a0512607bb9b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.064577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5717803e-8d1e-428a-b81b-a0512607bb9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.083721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5717803e-8d1e-428a-b81b-a0512607bb9b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:46.084913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5717803e-8d1e-428a-b81b-a0512607bb9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:46.097508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.098142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.098425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.098696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.099550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:46.099889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.101306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.137729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 252/507] 10.222.0.42 () {56 vars in 1137 bytes} [Tue Jul 26 02:57:45 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:46.138495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.148398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.156614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 256/508] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:46.157669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5717803e-8d1e-428a-b81b-a0512607bb9b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.223361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.254816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae99c7e-ca15-4c5d-aa78-07f7dd7830e5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.264089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5717803e-8d1e-428a-b81b-a0512607bb9b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.289705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.290351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.290623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.290893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.298646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.299351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.299636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.299911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.300920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.302403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.310087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.311574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:46.316975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:46.317967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:46.318838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:46.318838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:46.318838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.318838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:46.318838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 253/509] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:46.319120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:46.328136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:46.329069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:46.329915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:46.329915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:46.329915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.329915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:46.329915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 257/510] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:46.330170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.378140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.425008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6383f0-bfe1-43ae-a175-5f5db20b0667 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.428790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94c525b-6148-492c-a701-16eb249eda49 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.447569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.458822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.459479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.459759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.460033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.460792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:46.461101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.461806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.462427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.462516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.462704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.463000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.475479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 254/511] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:46.476584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.529605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.531385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.579157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110bd2cd-ffe4-4ca5-a943-a2eae7fef162 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.583466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.613065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.613710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.613984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.614256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.624495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.625959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.636507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 258/512] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:46.637417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:46.641463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:46.642357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:46.643235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:46.643235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:46.643235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.643235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:46.643235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 255/513] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:46.643512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.679164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.740697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f931910-958b-4a5b-9e6e-91e634668825 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.745459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f6223-8f39-47e1-8465-8103b1417b81 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.775275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.775920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.776199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.776475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.777187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:46.777492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.779086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.779448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.780085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.780364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.780643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.781343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:46.781699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.783070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.791000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 259/514] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:46.791802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.800995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 256/515] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:46.802298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.803622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.890962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef606fc-802a-437a-bb79-8091900d00f7 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.897057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:46.903067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee5428a-462e-43c0-8f90-cbdb1fd742a6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:46.925042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.925691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.925969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.926226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.933801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.935268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.936888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:46.937557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:46.937823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:46.938097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:46.946517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:46.947421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:46.948310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 260/516] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/users/cinder => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:46.949513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:46.949680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:46.949988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:46.966693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:46.967637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:46.968475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:46.968475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:46.968475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:46.968475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:46.968475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 257/517] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:46.968475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.025875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.061364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e165bf79-bfed-4cfd-887f-4340ef568f66 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.072058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f859e11-7c2d-48a8-ac71-375eef58146d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.102398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.103145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.103612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.104036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.104937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:47.105386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.107276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.107440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.108095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.108371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.108640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.117004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.118462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.119899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 261/518] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:47.120757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:47.133293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:47.134208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:47.135084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 258/519] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:47.135550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.147121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.233694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f88777-e5f0-4edf-b5b7-04fc7616c4d7 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.235794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fd5cdb-2f69-4ff5-8276-a356d3ba3a35 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.259664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.268047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.268695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.268978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.269250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.271889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.272518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.272793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.273064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.273874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:47.274182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.275660 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.276502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.277994 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:47.289623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:47.290509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:47.291426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 262/520] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/users/neutron => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:47.291970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.298452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 259/521] 10.222.0.42 () {56 vars in 1137 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:47.298452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.360284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.389791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0d60e8-c3d4-4785-b903-aa78658a64c5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.393537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed99e8aa-4abd-411b-a9f1-7ffb53179364 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.423466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.424091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.424369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.424639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.425358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:47.425661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.427069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.427841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.428647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.428928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.429203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.430304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.431774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.439356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 263/522] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:47.440178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.456304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 260/523] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:57:47 2022] POST /identity/v3/users => generated 247 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:57:47.457162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.517536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.573923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb7908a-76ce-4d51-9cf6-f4f5ee7e4c8b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.576473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e6cfdf-7af7-42b2-b867-2c1fbca90791 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.601890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.616686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.616686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.616947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.617239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.617333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.617852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.618171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.618396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.619195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:47.619498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.621004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.642070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 264/524] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:47.644419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.699160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.700702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.759390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9418531-9f15-4af7-8726-becc48aee696 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.769458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.795801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.796433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.796706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.796989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.797803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:47.798116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.799639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.815001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 261/525] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:47.815001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.819939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 265/526] 10.222.0.42 () {56 vars in 1139 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:47.824430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:47.839876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:47.931193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49a0564-72f5-4b85-8f7f-6f5fcf08b55f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.935609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67797e6f-460a-4d2b-b573-eebd006dee8a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:47.970469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb87742-236d-4a20-93d0-839811056138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.971582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb87742-236d-4a20-93d0-839811056138 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.971949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb87742-236d-4a20-93d0-839811056138 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.972220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb87742-236d-4a20-93d0-839811056138 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.975008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:47.975638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:47.975917 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:47.976193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:47.977003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=e2c09978fa3f44d895cbc17efdca2b32) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:47.977314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.978783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.980764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb87742-236d-4a20-93d0-839811056138 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:47.982382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb87742-236d-4a20-93d0-839811056138 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:47.985263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 266/527] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/role_assignments?user.id=e2c09978fa3f44d895cbc17efdca2b32&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:47.986398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-acb87742-236d-4a20-93d0-839811056138 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:48.019994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:48.020903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:48.021773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 262/528] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:48.022764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb87742-236d-4a20-93d0-839811056138 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.051105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:48.127252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:48.150817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb87742-236d-4a20-93d0-839811056138 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:48.159893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2151b0-ad5e-4175-99d7-e3998a27262d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:48.208506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:48.209208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:48.209628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:48.210026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:48.212569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:48.213487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:48.213897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:48.214296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:48.225912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:48.228384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:48.249378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:48.250296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:48.251169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:48.251169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:48.251169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:48.251169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:48.252575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 267/529] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:48.253540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.314825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:48.316262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:48.354749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:48.385591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daa8b1d-cad5-4a2d-9b40-da4567280176 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:48.404380 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:48.425718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 263/530] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:48.426674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.432879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:48.433891 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:48.434372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:48.434849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:48.435944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:48.436494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:48.438290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:48.453025 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 268/531] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:48.454314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.547729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e67e21-276f-462f-90c9-a45faec4b838 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:48.572407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d383a05-6679-4770-a6db-291b7e5132b0 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:48.596197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:48.597261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:48.597538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:48.597808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:48.598531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:48.598831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:48.600946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:48.618214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 264/532] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:48.618809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:48.619247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.619504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:48.619787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:48.620062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:48.634967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:48.637326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:48.640659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:48.664682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:48.665822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:48.666854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:48.666854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:48.666854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:48.666854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:48.666854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 269/533] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:48.667987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.689940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:48.765065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33c2143-9815-45fc-b938-07ac74d50153 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:48.802209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:48.802807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:48.803106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:48.803382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:48.804185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=81ec7fc0b48643739402efba1285c713) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:48.804501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:48.805997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:48.818215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112a4cac-755b-4fd8-bb50-0d901efe2107 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:48.819260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 265/534] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/role_assignments?user.id=81ec7fc0b48643739402efba1285c713&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:48.820291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.864494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:48.865633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:48.866109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:48.866575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:48.876029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:48.877971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:48.894012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:48.894938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:48.895805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:48.898705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 270/535] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:48.909667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:48.994225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:49.062003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fadfb61-bb54-406f-beb8-7ddbf33dd900 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:49.070862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d92b7d-dc39-49e5-9d4c-4fa8e522cfae None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:49.075206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:49.154163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:49.155171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:49.155487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:49.155783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:49.156886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:49.157247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:49.158942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:49.175162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:49.175815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:49.176089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:49.176432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:49.177347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:49.177703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:49.179582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:49.187786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 266/536] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 368 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:49.193722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:49.198053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 271/537] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:49.204745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:49.420074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6269b42c-a723-4942-9404-7aed596731e8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:49.464338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:49.498831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d7ba9b-b53b-45b6-ba35-0aa7f4906a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:49.499942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d7ba9b-b53b-45b6-ba35-0aa7f4906a97 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:49.500440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d7ba9b-b53b-45b6-ba35-0aa7f4906a97 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:49.501709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d7ba9b-b53b-45b6-ba35-0aa7f4906a97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:49.503920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d7ba9b-b53b-45b6-ba35-0aa7f4906a97 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=48f06347846c48c2933cee7ebd32b947) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:49.504497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d7ba9b-b53b-45b6-ba35-0aa7f4906a97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:49.507614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d7ba9b-b53b-45b6-ba35-0aa7f4906a97 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:49.520695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 272/538] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:49 2022] GET /identity/v3/role_assignments?user.id=48f06347846c48c2933cee7ebd32b947&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 259 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:49.569492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7341fe-2e6c-4117-a53a-66c58c62f6c6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:49.624489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:49.625450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:49.625876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:49.626291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:49.640542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:49.642375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:49.661322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:49.662408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:49.663700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:49.667393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6b9fdc-3144-431f-a0d2-b42f7efbc8af None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:49.667393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6b9fdc-3144-431f-a0d2-b42f7efbc8af None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:49.667393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6b9fdc-3144-431f-a0d2-b42f7efbc8af None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:49.667393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 273/539] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:57:49.669550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 267/540] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 476 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:49.672751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a6c02-bb4d-4c63-a928-1b73e0937eb6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:49.672751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a6c02-bb4d-4c63-a928-1b73e0937eb6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:49.672751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784a6c02-bb4d-4c63-a928-1b73e0937eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:49.675926 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:49.692591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-784a6c02-bb4d-4c63-a928-1b73e0937eb6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:49.693850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-784a6c02-bb4d-4c63-a928-1b73e0937eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:49.717257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:49.846319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 274/541] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:49.886418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784a6c02-bb4d-4c63-a928-1b73e0937eb6 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:49.908830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47e576-3a1d-4ddb-ab8b-47687d5e86cb None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:49.950737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:49.951652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:49.952205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:49.952611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:49.953516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:49.953984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:49.955653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:49.976367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 268/542] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:49.983686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:50.085799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784a6c02-bb4d-4c63-a928-1b73e0937eb6 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:50.097473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:50.127094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:50.127761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:50.128628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:50.128909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:50.141710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:50.147293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:50.148276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:50.149067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:50.149839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 275/543] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:50.160162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:50.178811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:50.204914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6751d269-30e6-427d-b478-1afc7301f5b5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:50.273727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dd030b-5962-46b7-bafc-62f92b3e17b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:50.274533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd030b-5962-46b7-bafc-62f92b3e17b5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:50.274817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd030b-5962-46b7-bafc-62f92b3e17b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:50.276441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dd030b-5962-46b7-bafc-62f92b3e17b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:50.278133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dd030b-5962-46b7-bafc-62f92b3e17b5 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=29a1868eb95f4396aa3fd2da677b40f1) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:50.278444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dd030b-5962-46b7-bafc-62f92b3e17b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:50.280745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dd030b-5962-46b7-bafc-62f92b3e17b5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:50.289788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 269/544] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:49 2022] GET /identity/v3/role_assignments?user.id=29a1868eb95f4396aa3fd2da677b40f1&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 259 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:50.298202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b528fb46-825f-4dad-ac7a-38f1f20924e0 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:50.298562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b528fb46-825f-4dad-ac7a-38f1f20924e0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:50.298840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b528fb46-825f-4dad-ac7a-38f1f20924e0 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:50.300739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 270/545] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:50 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:50.308163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda40b9a-0638-4414-93e2-e112aa547e61 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:50.308435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda40b9a-0638-4414-93e2-e112aa547e61 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:50.308712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda40b9a-0638-4414-93e2-e112aa547e61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:50.341115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-fda40b9a-0638-4414-93e2-e112aa547e61 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:50.341606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d05da0-40ed-46f0-9782-185a4f12a20d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:50.342316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fda40b9a-0638-4414-93e2-e112aa547e61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:50.406500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:50.407491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:50.407948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:50.408421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:50.409335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:50.409815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:50.411434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:50.426004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 271/546] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:50.431175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 276/547] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:50.438184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda40b9a-0638-4414-93e2-e112aa547e61 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:50.448974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:50.520105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:50.587778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:50.623649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda40b9a-0638-4414-93e2-e112aa547e61 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:50.643112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3dd0c-24e4-4277-b01f-07de2e1f6e66 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:50.697569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:50.699350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:50.700063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:50.701687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:50.716453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:50.717523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:50.717998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:50.718466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:50.723071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:50.725774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:50.743205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:50.749153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:50.750051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:50.750827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:50.763334 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 277/548] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:50.764850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:50.765722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:50.767226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:50.767226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:50.767226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:50.767226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:50.769583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 272/549] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:50.775007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:50.775175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:50.939579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853bc59-ef0a-4536-9abb-66d9ad939d02 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:50.950734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:50.977257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f112ea8e-8059-49e6-b891-9d1dd6e62134 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:51.004581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:51.019178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.019178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.019178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.019178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.019551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:51.019551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.021417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.047479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 273/550] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:51.054735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:51.070620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.071288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.071566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.071840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.072572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:51.072876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.074271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.086828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 278/551] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:51.095532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:51.178977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac8772c-77d3-4e03-a4a2-df823fb19b16 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:51.212573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.213241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.213514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.213785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] PATH_INFO: `/v3/users/swift` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.223974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.225454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:51.238702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:51.239785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:51.241226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 274/552] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:51.247912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:51.273371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb8c6ee-7a58-40f5-a9e0-dbd2390a8e84 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:51.324693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de3d661-781a-4931-be3b-bd952e647d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.325937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de3d661-781a-4931-be3b-bd952e647d0d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.326635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de3d661-781a-4931-be3b-bd952e647d0d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.327586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de3d661-781a-4931-be3b-bd952e647d0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.353033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:51.360956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de3d661-781a-4931-be3b-bd952e647d0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.365451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de3d661-781a-4931-be3b-bd952e647d0d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7de3d661-781a-4931-be3b-bd952e647d0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:51.386924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:51.389105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 279/553] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2580fc8-4239-440d-a46c-a3e91cc0512d None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2580fc8-4239-440d-a46c-a3e91cc0512d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.391163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2580fc8-4239-440d-a46c-a3e91cc0512d None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.396629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 280/554] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:51 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:51.397528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2580fc8-4239-440d-a46c-a3e91cc0512d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:51.410018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:51.427689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117d9460-061d-4196-b37d-f5614df0f70f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:51.493325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9f3036-e7d3-493a-b194-3ac63c136335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.494112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9f3036-e7d3-493a-b194-3ac63c136335 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.494395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9f3036-e7d3-493a-b194-3ac63c136335 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.494666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9f3036-e7d3-493a-b194-3ac63c136335 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.495518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9f3036-e7d3-493a-b194-3ac63c136335 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:51.495827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9f3036-e7d3-493a-b194-3ac63c136335 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.497531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9f3036-e7d3-493a-b194-3ac63c136335 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.520972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 275/555] 10.222.0.42 () {56 vars in 1135 bytes} [Tue Jul 26 02:57:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:51.522535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d175417d-6f56-4dd7-a4c8-12fb42e22817 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.522815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d175417d-6f56-4dd7-a4c8-12fb42e22817 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.523112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d175417d-6f56-4dd7-a4c8-12fb42e22817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.536197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2580fc8-4239-440d-a46c-a3e91cc0512d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:51.544288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d175417d-6f56-4dd7-a4c8-12fb42e22817 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:51.545627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d175417d-6f56-4dd7-a4c8-12fb42e22817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:51.572722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.573435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.573721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.574001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.574832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:51.575126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.576557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.593863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 281/556] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:51.595231 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:51.635427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 276/557] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:51.636663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d175417d-6f56-4dd7-a4c8-12fb42e22817 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:51.671716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:51.736699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:51.746351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382f1c5-ad0f-4c3e-874b-0299f24d8434 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:51.793130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05db836e-0506-4bda-a7ac-93922573e747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.793888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db836e-0506-4bda-a7ac-93922573e747 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.794360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db836e-0506-4bda-a7ac-93922573e747 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.794673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05db836e-0506-4bda-a7ac-93922573e747 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.824602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d175417d-6f56-4dd7-a4c8-12fb42e22817 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:51.871905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:51.872627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:51.872909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:51.873221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] PATH_INFO: `/v3/users/nova` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:51.881166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.882639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.892797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05db836e-0506-4bda-a7ac-93922573e747 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:51.894265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05db836e-0506-4bda-a7ac-93922573e747 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:51.894705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:51.897528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 277/558] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:57:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:51.899400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:51.973475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.019072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.022054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff51707-d605-4890-92e5-654c1a4eb5ac None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.039362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 282/559] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:52.041647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2688e505-229c-4252-bc7c-db62c3f63924 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.042107 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2688e505-229c-4252-bc7c-db62c3f63924 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.042589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2688e505-229c-4252-bc7c-db62c3f63924 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.045494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 283/560] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:52.046825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2688e505-229c-4252-bc7c-db62c3f63924 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.074149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.075362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.075933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.076412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.098430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:52.108382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:52.109549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:52.110377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 278/561] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:52.111124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.195376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2688e505-229c-4252-bc7c-db62c3f63924 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.250344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9ebe2-19cd-4f28-ac33-415edc8dc55d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.251341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9ebe2-19cd-4f28-ac33-415edc8dc55d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.251892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9ebe2-19cd-4f28-ac33-415edc8dc55d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.252298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9ebe2-19cd-4f28-ac33-415edc8dc55d None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.253241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9ebe2-19cd-4f28-ac33-415edc8dc55d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:52.254162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9ebe2-19cd-4f28-ac33-415edc8dc55d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:52.255905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9ebe2-19cd-4f28-ac33-415edc8dc55d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:52.259553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5380ce-756e-4ab1-b170-87e42d559391 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.279734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 284/562] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:52.281350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02add2-104f-4df0-b630-b1880c889f6e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.281637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02add2-104f-4df0-b630-b1880c889f6e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.281922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a02add2-104f-4df0-b630-b1880c889f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.301544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3a02add2-104f-4df0-b630-b1880c889f6e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:52.302766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3a02add2-104f-4df0-b630-b1880c889f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:52.320806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.322599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.323585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.323863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.324140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.335159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:52.336635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:52.352952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:52.354079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:52.355189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:52.355189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:52.355189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:52.355189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:52.358858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 279/563] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:52.359776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.376318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.384259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 285/564] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:52.385762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02add2-104f-4df0-b630-b1880c889f6e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.486986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60df30d-696e-4392-81b5-6972a54af27a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.492734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a02add2-104f-4df0-b630-b1880c889f6e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.524452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.525238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.525593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.525932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.526727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:52.527180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:52.528693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:52.530101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.530718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.531020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.531291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.542530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:52.542905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 280/565] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:52.544033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.544139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:52.561237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:52.562639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:52.564042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:52.564042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:52.564042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:52.564042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:52.567082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 286/566] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:52.567974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.573219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.639007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.666690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c905c-38cb-4610-aed8-7e14f98ec443 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.677208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60824fbc-17d6-4f20-b214-7a91f24089fc None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.699832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.700459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.700737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.701011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.701712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:52.702017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:52.703515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:52.710546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.710813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.711130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.711403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.715223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 281/567] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:52.716022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.722109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:52.727512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:52.728441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:52.729336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 287/568] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:52.730156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.789431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.817638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888eb1ac-4323-405f-8d3f-cf4f2089c0b9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.830381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a591a02-5013-4d05-b7ca-ac18d8f6b455 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.851821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.852474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.852758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.853028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.854984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:52.864093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcff7e7-f82a-4b07-909e-786b19bb4668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:52.865715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcff7e7-f82a-4b07-909e-786b19bb4668 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.865988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcff7e7-f82a-4b07-909e-786b19bb4668 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.866146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:52.866241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcff7e7-f82a-4b07-909e-786b19bb4668 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.866979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcff7e7-f82a-4b07-909e-786b19bb4668 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:52.867298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcff7e7-f82a-4b07-909e-786b19bb4668 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:52.867622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:52.868680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcff7e7-f82a-4b07-909e-786b19bb4668 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:52.880489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 288/569] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:52.881762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb1e2b4-ac97-4a52-9229-6d11db5c552e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:52.882057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb1e2b4-ac97-4a52-9229-6d11db5c552e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:52.882337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb1e2b4-ac97-4a52-9229-6d11db5c552e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:52.882461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:52.883341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:52.889424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:52.889424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:52.889424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:52.889424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:52.889424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 282/570] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:52.889424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.889759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 289/571] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:52.889759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb1e2b4-ac97-4a52-9229-6d11db5c552e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:52.987278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b549c90-f380-4270-a2b2-8f370e64ec2e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:52.998419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb1e2b4-ac97-4a52-9229-6d11db5c552e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.023478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.025205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.025834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.026359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.026359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.033323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.034570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.034785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.035232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.035510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.035778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.036503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:53.036802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.038153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:53.046235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:53.047134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:53.048001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:53.048414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 290/572] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:53.048414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.048535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 283/573] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:53.048535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.098307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.148975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0b6ad6-21b8-4094-a467-ba48b1c12e79 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.153699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e27b70-7765-4842-937e-1be39da88721 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.182942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.183617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.183931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.184201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.188076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39648e2f-4b9b-4383-839e-955aa52bc28b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.188703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39648e2f-4b9b-4383-839e-955aa52bc28b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.188984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39648e2f-4b9b-4383-839e-955aa52bc28b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.189255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39648e2f-4b9b-4383-839e-955aa52bc28b None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.189955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39648e2f-4b9b-4383-839e-955aa52bc28b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:53.190258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39648e2f-4b9b-4383-839e-955aa52bc28b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.190896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.191705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39648e2f-4b9b-4383-839e-955aa52bc28b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.192360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:53.203569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:53.204482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:53.205359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:53.205776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 284/574] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:53.205776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f3f350-9756-4e3d-aa39-5ffdf37f195f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.205776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f3f350-9756-4e3d-aa39-5ffdf37f195f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.205776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f3f350-9756-4e3d-aa39-5ffdf37f195f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.207243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 291/575] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:53.208379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.224254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-30f3f350-9756-4e3d-aa39-5ffdf37f195f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:53.225469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-30f3f350-9756-4e3d-aa39-5ffdf37f195f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:53.257789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.293785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 285/576] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:53.295001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f3f350-9756-4e3d-aa39-5ffdf37f195f None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.321178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d2124-f4f6-44f9-9a32-fd69fd36058a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.330579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.363915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.364611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.364868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.365223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.377567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.379096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:53.394007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:53.395704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:53.396730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:53.396730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:53.396730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:53.396730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:53.396730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 292/577] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:53.396730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.403223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f3f350-9756-4e3d-aa39-5ffdf37f195f None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.439141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.439783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.440063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.440335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.441104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:53.441428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.442880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.455542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 286/578] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:53.456068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.503526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.509140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a555806a-4231-4dca-ac12-eb0ffe97114d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.550432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.550846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.551118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.551455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.560994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.566090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.567582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604b5b60-4d97-43d8-b8ad-3cb5868cba4b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.568808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:53.581880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:53.582912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 293/579] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/users/glance => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:53.584889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.607379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.608001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.608277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.608543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.609297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:53.609594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.611061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.624681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 287/580] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:53.625572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.710811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf34a35-4cb1-4e26-8b9b-cb86c320f7a4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.731324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb25e8a-441a-4523-82b9-6bf6c81da828 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.756000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.756635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.756914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.757178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.767770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:53.768767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.769388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.769663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.769928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.770625 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:53.770920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.772342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:53.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:53.773962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:53.774785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 294/581] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:53.775518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.789251 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 288/582] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:53.790116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.798768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.809484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:53.886064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93441b15-dfb6-4f07-88ac-6ae2751df4e1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.894849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3882ef57-4ca4-4dc2-a00e-3713729f0b6f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:53.921015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.921645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.921921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.922189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.923016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=afebddca912546119eea61f568bbd60d) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:53.923323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.925032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.931390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 295/583] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/role_assignments?user.id=afebddca912546119eea61f568bbd60d&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:53.932778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:53.948199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:53.948856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:53.949134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:53.949405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:53.950249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:53.950554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:53.952127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:53.975236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 289/584] 10.222.0.42 () {56 vars in 1137 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:53.981637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.058003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:54.085029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a179b05a-85ed-4120-857b-992cf00e3824 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.095418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:54.126328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.127059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.127340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.127609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/e2c09978fa3f44d895cbc17efdca2b32/roles/262cff71501a4fd3b5d8663bc982bde7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.134859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632b19ff-721c-4f74-9909-938b0a036054 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.144633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=e2c09978fa3f44d895cbc17efdca2b32, role_id=262cff71501a4fd3b5d8663bc982bde7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.146021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.162922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 296/585] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:53 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/e2c09978fa3f44d895cbc17efdca2b32/roles/262cff71501a4fd3b5d8663bc982bde7 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:54.164146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.193995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.194832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.195200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.195421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.196187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:54.196490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.197948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.213017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 290/586] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:54.214073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.318679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4134e038-ba06-4b21-be17-9cd7b8e127ff None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.362498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:54.370446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.371211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.371503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.371805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.382065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.383597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.385116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc655c6-7853-4814-9659-57166dcd0d34 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:54.397843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:54.399014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 297/587] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/users/cinder => generated 84 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:54.400140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.432382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.433113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.433829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.434152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.438186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:54.452809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.454316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:54.472668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:54.473789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:54.474866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:54.474866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:54.474866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:54.474866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:54.474866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 291/588] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:54.476277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.550255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf75bb7e-d267-4ab0-bd17-52d707dbdad5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.605829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.606444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.607620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.607620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.622301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.624712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.635012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755394b2-b774-4cef-9f58-3f8fd7d27af1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:54.653929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:54.655047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:54.656263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:54.656263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:54.656263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:54.656263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:54.658930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 298/589] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:54.660010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.688933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.689693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.690041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.690386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.691318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:54.691715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.693356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.721357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 292/590] 10.222.0.42 () {56 vars in 1137 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:54.721357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.731321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:54.789391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:54.803816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a3a838-1b7c-4569-905e-b1531d15081d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.849515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.851009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.851462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.852122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.852978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:54.853436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.854290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7d107d-92f5-4879-984c-769036b1cdc0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:54.855179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.874462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 299/591] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:54.875999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:54.898128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:54.899149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:54.899570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:54.899944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:54.901017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:54.901418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:54.903534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:54.919394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 293/592] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:54.923891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.041232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8785ad6d-d9b5-4aa7-b762-15a0f6618c7d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.045919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:55.067718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52aeb1e-2ac6-4415-85f4-f7c7233c71da None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.077725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:55.095302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.096703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.097897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.098389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.115062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.116967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.117510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.118374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.118791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.119179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.128754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.131090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:55.137080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:55.138014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:55.138892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:55.138892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:55.138892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:55.138892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:55.145441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 300/593] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:55.145748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:55.146790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:55.147940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:55.149278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.149782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 294/594] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:55.150740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.300064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd10dd26-c477-48be-9ee7-b3df8051ca5e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.312929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0484b99e-9b7b-464d-90d1-6508b221ff6b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.365465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.366989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.367503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.367997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.368976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:55.369866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.372163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.380030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.381145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.382411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.382903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.388909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 295/595] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:55.390751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.406878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.410602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.424728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:55.436358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:55.439118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 26 02:57:55.440485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:55.442447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 301/596] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:55 2022] GET /identity/v3/users/neutron => generated 85 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:55.443482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.468205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:55.525422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557c1202-170b-4b92-a31a-e8dbd630000f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.564153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.564603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df65087-8bd0-459e-aa42-1c5784cac78b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.564781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.565066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.565341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.566115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:55.566424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.568068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.581537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 296/597] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:55.582521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.605593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.606736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.607263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.607728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.616011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.617981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:55.631469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:55.632409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:55.633378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:55.635753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 302/598] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:55.637050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.682606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:55.699742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc9422e-17a0-42e9-be72-898a63538a43 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.732559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:55.740180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.740823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.741178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.741368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.742240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:55.742548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.744180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.762145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925a67a9-683b-4f6e-bf49-a59ac4527242 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.769329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 297/599] 10.222.0.42 () {56 vars in 1139 bytes} [Tue Jul 26 02:57:55 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:55.770334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.803200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.804357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.805057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.805615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/81ec7fc0b48643739402efba1285c713/roles/262cff71501a4fd3b5d8663bc982bde7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.823920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=81ec7fc0b48643739402efba1285c713, role_id=262cff71501a4fd3b5d8663bc982bde7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.825824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.846855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 303/600] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:55 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/81ec7fc0b48643739402efba1285c713/roles/262cff71501a4fd3b5d8663bc982bde7 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:55.848480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.887591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a132d2-f20b-4b34-b95e-d84181d83d41 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:55.927951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:55.928611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:55.928889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:55.929161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:55.929946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:55.930258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:55.931902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:55.945058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 298/601] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:55.951487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:55.955487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:55.997668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0362724-0dd1-4a20-af2e-2e99ca7d7c39 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:56.035058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:56.059431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:56.060613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.062014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.062666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.084606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:56.088489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:56.112534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656524b5-8ee8-4fe0-94a7-4be64e6cacce None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:56.124821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:56.126106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:56.127010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:56.127010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:56.127010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:56.127010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:56.127010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 304/602] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:56.133499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:56.165297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ed0669-8371-46fd-a76f-2244750f237a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:56.167714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed0669-8371-46fd-a76f-2244750f237a None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.168662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed0669-8371-46fd-a76f-2244750f237a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.169414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ed0669-8371-46fd-a76f-2244750f237a None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/48f06347846c48c2933cee7ebd32b947/roles/262cff71501a4fd3b5d8663bc982bde7` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.193723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ed0669-8371-46fd-a76f-2244750f237a None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=48f06347846c48c2933cee7ebd32b947, role_id=262cff71501a4fd3b5d8663bc982bde7)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:56.196599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ed0669-8371-46fd-a76f-2244750f237a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:56.227012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 299/603] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:55 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/48f06347846c48c2933cee7ebd32b947/roles/262cff71501a4fd3b5d8663bc982bde7 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:56.227012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034bba7-5675-4d06-b1e9-18ad2b607b13 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.227012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034bba7-5675-4d06-b1e9-18ad2b607b13 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.227012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c034bba7-5675-4d06-b1e9-18ad2b607b13 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.228202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 300/604] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:56.235150 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a24020-c6d9-41fc-b3f5-f76f27883333 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.235568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a24020-c6d9-41fc-b3f5-f76f27883333 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.235846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a24020-c6d9-41fc-b3f5-f76f27883333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.256898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-10a24020-c6d9-41fc-b3f5-f76f27883333 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:56.258223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-10a24020-c6d9-41fc-b3f5-f76f27883333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:56.279146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425c752c-dfdf-4444-8f30-4084df7ccfd8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:56.321350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:56.325714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:56.326415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.326696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.326990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.327744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:56.328105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:56.329871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:56.344223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 301/605] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:56.346561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd7351e-567d-482b-906b-9ca26b2a3f8c None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.347094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd7351e-567d-482b-906b-9ca26b2a3f8c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.347572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd7351e-567d-482b-906b-9ca26b2a3f8c None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.349247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 302/606] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:56.358978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 305/607] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:56.359198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eb0a02-7fb0-4192-acae-4ac1024b2e60 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.359198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eb0a02-7fb0-4192-acae-4ac1024b2e60 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.359198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0eb0a02-7fb0-4192-acae-4ac1024b2e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.368238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:56.377097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f0eb0a02-7fb0-4192-acae-4ac1024b2e60 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:56.379057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f0eb0a02-7fb0-4192-acae-4ac1024b2e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:56.424061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:56.484436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 303/608] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:57:56.485341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0eb0a02-7fb0-4192-acae-4ac1024b2e60 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:56.514912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a7099b-94b0-48f9-8281-64f9f631f042 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:56.566766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:56.567475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.567757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.568459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.578773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:56.580635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:56.597291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:56.598507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:56.599659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 306/609] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:56.600145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:56.637952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0eb0a02-7fb0-4192-acae-4ac1024b2e60 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:56.668280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:56.700437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:56.701979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.702639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.703219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.722370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:56.731179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:56.732808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:56.734620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:56.738581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:56.740118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 304/610] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:56.742022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:56.756644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6ff4f-52a5-4c92-9500-c61045db8119 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:56.832182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:56.833898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.834828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.835770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.857961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:56.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:56.871958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:56.873461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:56.877567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 307/611] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:56.879458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:56.888245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8367b9-a4f7-4236-a668-1ff4437eec4a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:56.957969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:56.959553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:56.960274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:56.960999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:56.962615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:56.963460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:56.966336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:56.988887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 305/612] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:57:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:56.990759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:57.063708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:57.080121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ea895f-5127-4479-ad3d-86335132ccf8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:57.144907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:57.147187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:57.148091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:57.148810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:57.150518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:57.151347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:57.154587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:57.170271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 308/613] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:57.172276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:57.187384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd9eedb-d38a-49e3-8075-40ee2ad21bc2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:57.214920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:57.249839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:57.251408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:57.252175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:57.252906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:57.254790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:57.255602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:57.258538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:57.274905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 306/614] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:57.276791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:57.374574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f8b673-3e65-4d5e-95d1-d535535d44c2 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:57.433776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:57.435551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:57.436464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:57.437178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/29a1868eb95f4396aa3fd2da677b40f1/roles/262cff71501a4fd3b5d8663bc982bde7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:57.459554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:57.466615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=29a1868eb95f4396aa3fd2da677b40f1, role_id=262cff71501a4fd3b5d8663bc982bde7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:57.470015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0399ce92-3125-42aa-8a75-fe7d237f93f0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:57.471083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:57.495285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 309/615] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:57:57 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/29a1868eb95f4396aa3fd2da677b40f1/roles/262cff71501a4fd3b5d8663bc982bde7 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:57:57.497537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:57.580528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:57.581212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:57.581488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:57.581767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:57.612239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:57.613731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:57.648681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:57.650491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:57.652330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:57.652330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:57.652330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:57.652330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:57.652330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 307/616] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 374 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:57.655651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:57.659232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:57.730342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab23955-e5cb-47e9-8c46-6bf3440ab639 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:57.799882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:57.801506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:57.802184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:57.802862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:57.822897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:57.825943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:57.865002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:57.866069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:57.867075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:57.867075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:57.867075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:57.867075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:57.867075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 310/617] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:57.875011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:57.899676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca4dbda-0491-4cd4-b859-e7e609052c97 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:57.982029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:57.982914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:57.984261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:57.985363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:57.986528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:57.988165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:57.989743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:58.017981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 308/618] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 363 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:58.022176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:58.040118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:58.076838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d04057d-429e-4881-8dbe-70813f5c2b8c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:58.123316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:58.146360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:58.148144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.148427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.148704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.151998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:58.152318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:58.153758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:58.186811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 311/619] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:58.191879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:58.233249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909718c4-d381-4ced-a768-96ca7ac95e6e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:58.305796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:58.306573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.306879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.307233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.337640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:58.340139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:58.368834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:58.370245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:57:58.371510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:58.379057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 309/620] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:57:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:58.385866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:58.409270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d703c6-c28d-438a-9cd9-e058b041db02 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:58.452626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e94452-9b26-4ce9-8295-311cab87b7ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:58.453293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e94452-9b26-4ce9-8295-311cab87b7ec None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.453569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e94452-9b26-4ce9-8295-311cab87b7ec None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.453835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e94452-9b26-4ce9-8295-311cab87b7ec None admin] PATH_INFO: `/v3/users/nova` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.464649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e94452-9b26-4ce9-8295-311cab87b7ec None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:58.466209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e94452-9b26-4ce9-8295-311cab87b7ec None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-d5e94452-9b26-4ce9-8295-311cab87b7ec None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:57:58.479827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:58.480920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:58.481849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 312/621] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:57:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:57:58.483327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1833bb-f9d7-4008-a603-8dd305326fe0 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.483327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1833bb-f9d7-4008-a603-8dd305326fe0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.483327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1833bb-f9d7-4008-a603-8dd305326fe0 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.489645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 313/622] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:58.489645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7f9f8-bbcf-4cf9-9439-7b8043e05ad5 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.489645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7f9f8-bbcf-4cf9-9439-7b8043e05ad5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.489645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f7f9f8-bbcf-4cf9-9439-7b8043e05ad5 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.489645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 314/623] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:57:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:57:58.489645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7f9f8-bbcf-4cf9-9439-7b8043e05ad5 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:58.513829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:58.544977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:58.554782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c10fbe9-d390-431a-8650-68c6b3f78128 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:58.593409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2ba540-f8b3-4044-aa6c-390466bd828d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:58.594178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ba540-f8b3-4044-aa6c-390466bd828d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.595033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ba540-f8b3-4044-aa6c-390466bd828d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.595452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ba540-f8b3-4044-aa6c-390466bd828d None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.596333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2ba540-f8b3-4044-aa6c-390466bd828d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:58.596706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2ba540-f8b3-4044-aa6c-390466bd828d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:58.598223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2ba540-f8b3-4044-aa6c-390466bd828d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:58.618309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 310/624] 10.222.0.42 () {56 vars in 1135 bytes} [Tue Jul 26 02:57:58 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:57:58.619199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f7f9f8-bbcf-4cf9-9439-7b8043e05ad5 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:58.619728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3852175-e979-48ad-8827-28dac1cf123a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.620116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3852175-e979-48ad-8827-28dac1cf123a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.620549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3852175-e979-48ad-8827-28dac1cf123a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.638749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-e3852175-e979-48ad-8827-28dac1cf123a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:58.640003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e3852175-e979-48ad-8827-28dac1cf123a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:58.665233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b7f9a3-631c-4b3d-b986-0a82d1ba0c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:58.666264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b7f9a3-631c-4b3d-b986-0a82d1ba0c9f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.666870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b7f9a3-631c-4b3d-b986-0a82d1ba0c9f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.667182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b7f9a3-631c-4b3d-b986-0a82d1ba0c9f None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.668690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b7f9a3-631c-4b3d-b986-0a82d1ba0c9f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:58.669203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b7f9a3-631c-4b3d-b986-0a82d1ba0c9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:58.671564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b7f9a3-631c-4b3d-b986-0a82d1ba0c9f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:58.691898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 315/625] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:57:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:58.693179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42c1211-5997-4dcb-a85f-ef44a30dce6e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.693459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42c1211-5997-4dcb-a85f-ef44a30dce6e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.693734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42c1211-5997-4dcb-a85f-ef44a30dce6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.712396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a42c1211-5997-4dcb-a85f-ef44a30dce6e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:57:58.713715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a42c1211-5997-4dcb-a85f-ef44a30dce6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:57:58.717665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 311/626] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:58.718702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3852175-e979-48ad-8827-28dac1cf123a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:58.787060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:58.832593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3852175-e979-48ad-8827-28dac1cf123a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:58.840700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 316/627] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:57:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:57:58.842370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42c1211-5997-4dcb-a85f-ef44a30dce6e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:58.878538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:58.879461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:58.879927 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:58.880401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:58.882780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:58.899958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:58.901443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:58.918779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:58.919910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:58.920835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:58.920835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:58.920835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:58.920835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:58.922425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 312/628] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:58.923675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:58.989162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42c1211-5997-4dcb-a85f-ef44a30dce6e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.028550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.029188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.029472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.029744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.076115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73950c8-0ded-42bf-8afb-c99b11f89557 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.088228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:59.116789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:59.118497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:59.118630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.119323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.119599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.119873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.136537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:59.142210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:59.143276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 313/629] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:59.144696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.165454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:59.253033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 317/630] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:57:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:57:59.254036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.264521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff83c77-f832-46b5-8513-1b55afe9fd07 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.304646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.305594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.306013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.306486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.318778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:57:59.324984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:59.326080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:57:59.327194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:59.329653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 314/631] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:59.330537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.366132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:59.377097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea307bb3-a119-45f3-9772-baac51fd1439 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.412095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:59.422872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.424484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.425273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.426021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.427386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:59.427915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:59.430477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:59.457263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 318/632] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:59.457469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b31969d-192e-4a68-bc6a-6dd65f4b6e94 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.457565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.518998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.520071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.520515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.520977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.522340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:59.522876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:59.525549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:59.542581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 315/633] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:59.544199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.580413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb9077f-a730-4da5-9604-8a2efd25f7eb None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.618685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.619375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.619658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.619933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.631949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:59.633439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:59.654584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:59.655604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:59.656504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:59.656504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:59.656504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:59.656504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:59.656504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 319/634] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:59.656811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.663999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:59.675380 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48efb5c8-260c-4a38-8f56-52c41dcd0535 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.705921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:59.716904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.717281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.717552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.717823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.718555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:57:59.718862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:59.720703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:59.730828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 316/635] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:57:59.731481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.778114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7b046f-a2da-4bd7-a73c-22518ff7fc7a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.816752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.817920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.818293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.818568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.826791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:59.828363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:59.840860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c3a781-1aa6-4102-bbee-d350ad11c229 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:57:59.842021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:59.843033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 320/636] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:59.844687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.881299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:57:59.881956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:57:59.882238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:57:59.882515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:57:59.894038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:57:59.895892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:57:59.911241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:57:59.913230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:57:59.915293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:57:59.915293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:57:59.915293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:57:59.915293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:57:59.917267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 317/637] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:57:59.918120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:57:59.922022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:57:59.958596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5bc392-26c2-4f6c-9b8c-238895adc238 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:57:59.960427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:00.000476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dd3b25-7947-4c53-88c9-b45dc9e68446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.001102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dd3b25-7947-4c53-88c9-b45dc9e68446 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.001379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dd3b25-7947-4c53-88c9-b45dc9e68446 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.001649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dd3b25-7947-4c53-88c9-b45dc9e68446 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.002364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dd3b25-7947-4c53-88c9-b45dc9e68446 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:00.002665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dd3b25-7947-4c53-88c9-b45dc9e68446 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.004094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1dd3b25-7947-4c53-88c9-b45dc9e68446 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.021300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 321/638] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:00.021300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aa6ef2-d5a6-4020-958f-640635b3fa49 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.021300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aa6ef2-d5a6-4020-958f-640635b3fa49 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.021300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69aa6ef2-d5a6-4020-958f-640635b3fa49 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.021300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 322/639] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:00.021300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aa6ef2-d5a6-4020-958f-640635b3fa49 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.036424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97c9a52-d29f-4c79-8590-0679833c851b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.073329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.074124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.074517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.074943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.090816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.092399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:00.109780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:00.111611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:00.113134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:00.113134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:00.113134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:00.113134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:00.113737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 318/640] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:57:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:00.115149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.141593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69aa6ef2-d5a6-4020-958f-640635b3fa49 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.179367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.180429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.180866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.181250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.181935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:00.182234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.183804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.190346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:00.197401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 323/641] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:00.198509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.225722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:00.248235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c814dc-68d3-4d0f-9d90-d30d10243c6b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.285607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540eecb8-113e-4627-83c4-0399c6f03d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.286329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540eecb8-113e-4627-83c4-0399c6f03d8a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.286582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540eecb8-113e-4627-83c4-0399c6f03d8a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.286861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540eecb8-113e-4627-83c4-0399c6f03d8a None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.287620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540eecb8-113e-4627-83c4-0399c6f03d8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:00.287923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540eecb8-113e-4627-83c4-0399c6f03d8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.289378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540eecb8-113e-4627-83c4-0399c6f03d8a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.310038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 319/642] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:00.310038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ed757-30b9-44fb-bc79-893bc1df0e2e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.310038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ed757-30b9-44fb-bc79-893bc1df0e2e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.310038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ed757-30b9-44fb-bc79-893bc1df0e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.321482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9e5456-796d-4f06-8bc1-254a2b104b4f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.326789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d21ed757-30b9-44fb-bc79-893bc1df0e2e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:00.328100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d21ed757-30b9-44fb-bc79-893bc1df0e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:00.360400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.361072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.361354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.361628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.369142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.370627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:00.382825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:00.383988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 324/643] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:00.384991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.415571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 320/644] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:00.416698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21ed757-30b9-44fb-bc79-893bc1df0e2e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.477038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:00.485523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:00.506455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600c7b14-56b9-4a7f-87b7-c4739e5a8951 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.546349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21ed757-30b9-44fb-bc79-893bc1df0e2e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.549418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.550423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.550825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.551309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.552535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:00.553024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.555555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.574241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 325/645] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:00.575569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.586361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.587165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.587515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.587861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/afebddca912546119eea61f568bbd60d/roles/262cff71501a4fd3b5d8663bc982bde7` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.609490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=afebddca912546119eea61f568bbd60d, role_id=262cff71501a4fd3b5d8663bc982bde7)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.612074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.627079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 321/646] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:58:00 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/afebddca912546119eea61f568bbd60d/roles/262cff71501a4fd3b5d8663bc982bde7 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:58:00.627979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.713115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e819d1c-28a0-4035-8377-7bb8b2b53126 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.762058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.763244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.763650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.763933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] PATH_INFO: `/v3/users/glance` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.777638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.778101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:00.779553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.796512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857c2b4-035a-4440-b804-71d8875a851d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:00.809938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:00.810987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 26 02:58:00.811933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:00.818805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 326/647] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/users/glance => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:00.818805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.823049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:00.865576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:00.866496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:00.866933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:00.867383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:00.868621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:00.869297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:00.871601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:00.886285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 322/648] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:00.887169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:00.983714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10aede6f-8140-4d5a-b7d5-d3c22a1f5faf None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.009612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79988eab-d56e-45f1-9fa0-ef1a6b9439c3 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.035319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.036361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.036858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.037321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.048968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:01.054770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:01.055977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:01.057009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:01.059494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 327/649] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:01.060554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.066445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.067359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.067782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.068217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.078228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:01.079976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:01.092536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:01.093615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:58:01.095179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:01.097205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 323/650] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:00 2022] GET /identity/v3/users/cinder => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:01.098164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.119928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:01.153435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:01.209341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689886f2-c142-47cc-b171-e5d519c6a58d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.231832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27df5a9-b724-416c-8265-4b218cf0efa4 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.266139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.266814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.267233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.267679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.269172 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:01.269597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:01.271002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:01.278446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.282180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.282462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.282755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.283604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=e2c09978fa3f44d895cbc17efdca2b32) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:01.283922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:01.285396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:01.298989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 328/651] 10.222.0.42 () {56 vars in 1137 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:01.298989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.299312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 324/652] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/role_assignments?user.id=e2c09978fa3f44d895cbc17efdca2b32&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:01.299312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.413698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:01.441422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf7a0fa-aa08-4a3a-9775-0472cbfb2696 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.445621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a502be-240d-4fcd-8909-2cda570cebda None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.474617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:01.496300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.496545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.497355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.497508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.497861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.497973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.498071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.498151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.498886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:01.499612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:01.500726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:01.508599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:01.508599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:01.513671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:01.524981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 329/653] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:01.528492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.529468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 325/654] 10.222.0.42 () {56 vars in 1137 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:01.532195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.666365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8583fc4d-10f1-4e39-8932-cb04b81f4fe7 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.685939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1188696-6cb8-4902-8588-584410a02545 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.705256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.705889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.706166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.706533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.732748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.733720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.734007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.734287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:01.744613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:01.750685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:01.752282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:01.765571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:01.773195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:01.774420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:01.775641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:01.775641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:01.775641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:01.775641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:01.775641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 326/655] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:01.779895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.794969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:01.796911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:01.921294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b2cfca-e94b-4786-a729-c3018c84142e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:01.955074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 330/656] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:58:01.956432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:01.966018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:01.966657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:01.966971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:01.967386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.053271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:02.067787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.069722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.076724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:02.103221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272b4bc6-366e-43fe-83ff-a1a1fcfedc15 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.149777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.150836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.151345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.151815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.152824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:02.153326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.155113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.169739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 331/657] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:02.171043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.215730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 327/658] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:58:02.216645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.298003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da9430d-2412-4668-9de7-bfae17c6adc6 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.340317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.341372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.341843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.342313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.353132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:02.356641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.358422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.360965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:02.368423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28a2764-2bbd-4353-b8b5-7bbb80f83c37 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:02.376869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:02.377845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:02.378740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:02.378740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:02.378740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:02.378740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:02.381042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 332/659] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:02.382051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.421600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a612017c-288b-4917-9009-596f6508e0de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.422266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612017c-288b-4917-9009-596f6508e0de None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.422535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612017c-288b-4917-9009-596f6508e0de None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.422803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a612017c-288b-4917-9009-596f6508e0de None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.433993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a612017c-288b-4917-9009-596f6508e0de None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.436432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a612017c-288b-4917-9009-596f6508e0de None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-a612017c-288b-4917-9009-596f6508e0de None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:02.453932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:02.454974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 26 02:58:02.455940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:02.461966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 328/660] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/users/neutron => generated 85 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:02.461966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a612017c-288b-4917-9009-596f6508e0de None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.505796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da136bd-5274-4079-8bf8-b74938c66897 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.541796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.542460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.542737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.543109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.556530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.558997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:02.581663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:02.582658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:02.583577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:02.583577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:02.583577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:02.583577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:02.584637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 333/661] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:02.585850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.608093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a612017c-288b-4917-9009-596f6508e0de None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.650459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:02.665902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.667066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.667676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.668439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.669694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:02.670220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.672511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.692834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 329/662] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:02.694281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.711089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:02.711640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e5ec8b-4baa-4d3a-88f3-acb94999c8fd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.746560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.747235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.747513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.747787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.748643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:02.748890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.750567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.773407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 334/663] 10.222.0.42 () {56 vars in 1139 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:02.773407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.814615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93289cd-2089-4df2-a772-f7a99ee13495 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.853360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.853991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.854271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.854540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.855265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:02.855631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.857084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.868786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 330/664] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:02.869620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.882250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bac7c-09a4-4a26-b0ea-081006fa84c4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:02.884987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:02.920977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a552aaf3-834a-481c-8cfe-5505ebabed8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:02.921632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a552aaf3-834a-481c-8cfe-5505ebabed8d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.921915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a552aaf3-834a-481c-8cfe-5505ebabed8d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.922182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a552aaf3-834a-481c-8cfe-5505ebabed8d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.929249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a552aaf3-834a-481c-8cfe-5505ebabed8d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:02.930711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a552aaf3-834a-481c-8cfe-5505ebabed8d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a552aaf3-834a-481c-8cfe-5505ebabed8d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:02.942889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:02.944094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:02.944996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 335/665] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:02.945976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef63a3-4d13-4320-8a8f-57863f6c3e46 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:02.945976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef63a3-4d13-4320-8a8f-57863f6c3e46 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:02.946144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ef63a3-4d13-4320-8a8f-57863f6c3e46 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:02.952925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 336/666] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:02.952925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef63a3-4d13-4320-8a8f-57863f6c3e46 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:02.961524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.001607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042db7d6-208d-405f-8588-b52762aad48e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.037901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b376cf31-dd14-4139-9eb2-539e80969e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.038991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b376cf31-dd14-4139-9eb2-539e80969e8c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.038991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b376cf31-dd14-4139-9eb2-539e80969e8c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.039187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b376cf31-dd14-4139-9eb2-539e80969e8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.072669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ef63a3-4d13-4320-8a8f-57863f6c3e46 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.112809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.113580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.114274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.114274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.115776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b376cf31-dd14-4139-9eb2-539e80969e8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.117377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b376cf31-dd14-4139-9eb2-539e80969e8c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.121448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.122904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:03.135462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:03.136461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:03.137385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 337/667] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:03.138545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.170342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.203656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.238784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 331/668] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:58:03.239947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c174fd22-5783-48f6-bb9e-0c3d4b3c1528 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.240228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c174fd22-5783-48f6-bb9e-0c3d4b3c1528 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.240501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c174fd22-5783-48f6-bb9e-0c3d4b3c1528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.258290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c174fd22-5783-48f6-bb9e-0c3d4b3c1528 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:03.259516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c174fd22-5783-48f6-bb9e-0c3d4b3c1528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:03.266080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93eb69-b1cf-44af-aaf0-d88b7c498d09 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.305075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.305727 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.306068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.306343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.307142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:03.307464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.308997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.327098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 338/669] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:03.327098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.338382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 332/670] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:03.347979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c174fd22-5783-48f6-bb9e-0c3d4b3c1528 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.413744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.430651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d961128-c45e-472c-8083-f81002757e94 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.466160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.468837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.469679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c174fd22-5783-48f6-bb9e-0c3d4b3c1528 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.469985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.470299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.470582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.471542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:03.471858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.473316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.486576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 339/671] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:03.487510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.516728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.517464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.517731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.518055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.529260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.530757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:03.546268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:03.547252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:03.548756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:03.548756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:03.548756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:03.548756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:03.548756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 333/672] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:03.548756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.597107 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c991959c-f5a2-47a1-a511-35aebdfbacdd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.630518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.631160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.631435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.631701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.632503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=81ec7fc0b48643739402efba1285c713) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:03.632816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.634232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.640196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 340/673] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/role_assignments?user.id=81ec7fc0b48643739402efba1285c713&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:03.641080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.653421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.660083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afb5fd6-9684-48df-bdd9-3072a818d765 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.701124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3c2469-9aca-4e7c-9ac0-c1b7342b4e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.701937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3c2469-9aca-4e7c-9ac0-c1b7342b4e40 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.702290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3c2469-9aca-4e7c-9ac0-c1b7342b4e40 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.702628 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3c2469-9aca-4e7c-9ac0-c1b7342b4e40 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.715202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3c2469-9aca-4e7c-9ac0-c1b7342b4e40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-2e3c2469-9aca-4e7c-9ac0-c1b7342b4e40 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:03.721148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:03.722381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:03.723343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 334/674] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:03.724104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222f4825-cbda-489a-b386-c614c2773284 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.724355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222f4825-cbda-489a-b386-c614c2773284 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.724631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222f4825-cbda-489a-b386-c614c2773284 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.726166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 335/675] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:03.726767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222f4825-cbda-489a-b386-c614c2773284 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.746164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.773890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5e5360-8ffa-4468-9ad4-cf167d71e9dd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.813724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.814373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.814652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.814928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.815822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=48f06347846c48c2933cee7ebd32b947) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:03.816136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.817651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.825249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 341/676] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/role_assignments?user.id=48f06347846c48c2933cee7ebd32b947&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:03.826139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:03.866154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222f4825-cbda-489a-b386-c614c2773284 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:03.911066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607ad582-de12-4b0f-9fde-7d5fe10fce3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:03.911812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607ad582-de12-4b0f-9fde-7d5fe10fce3c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.912098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607ad582-de12-4b0f-9fde-7d5fe10fce3c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.912387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607ad582-de12-4b0f-9fde-7d5fe10fce3c None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.913939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607ad582-de12-4b0f-9fde-7d5fe10fce3c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:03.913939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607ad582-de12-4b0f-9fde-7d5fe10fce3c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:03.915070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607ad582-de12-4b0f-9fde-7d5fe10fce3c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:03.927395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:03.930427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 336/677] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:03.931343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1760c2f-002a-4dd0-b11b-ddad3a72571c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:03.931563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1760c2f-002a-4dd0-b11b-ddad3a72571c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:03.931842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1760c2f-002a-4dd0-b11b-ddad3a72571c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:03.953848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f1760c2f-002a-4dd0-b11b-ddad3a72571c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:03.954898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f1760c2f-002a-4dd0-b11b-ddad3a72571c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:03.971970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0788b91-325a-4215-b8ec-90f6b03d2e8d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.017563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.018302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.018586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.018870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.019657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:04.019968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:04.021394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:04.035296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 342/678] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:58:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:04.036198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.049935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 337/679] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:04.051122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1760c2f-002a-4dd0-b11b-ddad3a72571c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.055386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:04.182892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb01e33-0007-4519-a40f-493a6f39ee7d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.186142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1760c2f-002a-4dd0-b11b-ddad3a72571c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.226991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.227755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.228120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.228405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.229638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.230996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.231934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.232410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.234764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:04.243249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:04.244487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:04.245347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:04.247436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:04.261024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:04.262069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:04.262973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:04.264455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 343/680] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:04.269411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:04.274932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:04.275886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:04.276800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:04.276800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:04.276800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:04.276800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:04.276800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 338/681] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:04.277590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.340691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:04.396204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611713d6-480e-4eda-8e83-f61c3e8de745 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.406459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c10c3b-b1ca-4c2b-b82c-012e65a4284b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.433189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.433745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.434030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.434306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.435111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:04.435563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:04.437085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:04.444780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.445797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.446275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.446744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.456041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 339/682] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:04.456041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.464488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:04.470641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:04.471708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:04.472548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:04.475304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 344/683] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:04.476408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.540059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:04.571615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963242fd-7683-4885-9511-6c81cae17fff None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.588610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:04.614122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.614668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.614968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.615286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.616341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:04.616501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:04.618021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:04.631339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 340/684] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:04.632219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.632478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d5ee5f-5940-4ec2-928d-c8edfabaadad None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.684376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.685347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.685844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.686337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.687548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=29a1868eb95f4396aa3fd2da677b40f1) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:04.688112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:04.689785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:04.703279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 345/685] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/role_assignments?user.id=29a1868eb95f4396aa3fd2da677b40f1&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 608 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:04.704848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.747631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09107330-8a38-4f07-9cb4-fabab60ee5a7 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.815895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.816768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.817054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.817331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:04.818979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:04.818979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:04.820019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:04.830117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 341/686] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:04.835716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:04.877299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:04.921984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f18bf23-b2a7-4e4a-9572-df1976bb476d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.952888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:04.967588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f2be13-7ffe-4f32-97a2-db2e7bdf8c97 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:04.984144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:04.985305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:04.985800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:04.986143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.002330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.004064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.011994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:05.012986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:05.013459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:05.013904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:05.024069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:05.025085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:05.026607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 346/687] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:05.031316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.032017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:05.032821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:05.050370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:05.051369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:05.052292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:05.052292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:05.052292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:05.052292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:05.052292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 342/688] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:05.057625 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:05.176502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:05.185685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f584ac-1083-40c4-80f0-1b9c48ee84c9 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:05.191590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d9cf84-c86a-42da-aeb1-d9806edbd7ed None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:05.228472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:05.229019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:05.229294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:05.229562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.230280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:05.230726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.232156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.233788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:05.235192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:05.235713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:05.236186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.237263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:05.237787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.239983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.246144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 343/689] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:05.251520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:05.261151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 347/690] 10.222.0.42 () {56 vars in 1135 bytes} [Tue Jul 26 02:58:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:05.272252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:05.357545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da506245-5e7f-4736-8b0d-3debdaefea38 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:05.365914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:05.395499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:05.396594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:05.397074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:05.397546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.405639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.407575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:05.420214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:05.421176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:05.422122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:05.424848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 344/691] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:58:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:05.431948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:05.450749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:05.500861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9499cf-799e-47f7-af8b-ce2095125732 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:05.565315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:05.565942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:05.566220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:05.566490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.605805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efd6d92-7774-4a46-b053-010e4d6b9011 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:05.641246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:05.641885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:05.642165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:05.642437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.643382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:05.643693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.645328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.664830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 345/692] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:58:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:05.670644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:05.681012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.682906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.734759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:05.779552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ad498d-a949-4825-95ed-d3fa34c96cc8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:05.783256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:05.816447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:05.817764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:05.818269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:05.818741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:05.870582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 348/693] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 599 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:58:05.875967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:05.890799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:05.892761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:05.959325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:05.995783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 346/694] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 26 02:58:06.001775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.069582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563a3409-d660-4b18-8f93-4a7ec402b1b7 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.104273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24af373-cf74-4d8a-83dc-d1701182b7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.104973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24af373-cf74-4d8a-83dc-d1701182b7a4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.105194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24af373-cf74-4d8a-83dc-d1701182b7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.105492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24af373-cf74-4d8a-83dc-d1701182b7a4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.108405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:06.112381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52df3d34-47c2-4fb0-8742-504d0e0f4885 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.116313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24af373-cf74-4d8a-83dc-d1701182b7a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.118420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24af373-cf74-4d8a-83dc-d1701182b7a4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a24af373-cf74-4d8a-83dc-d1701182b7a4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:06.133630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:06.134559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:06.135474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:06.135474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:06.135474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:06.135474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:06.135474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 349/695] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:06.136500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bc87ff-18e6-41ed-88a7-5cb8e7eff0bd None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.136500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bc87ff-18e6-41ed-88a7-5cb8e7eff0bd None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.136500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bc87ff-18e6-41ed-88a7-5cb8e7eff0bd None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.141781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 350/696] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:06.141781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c79d29-eee5-4399-bb87-6e2b40a2b2ba None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.141781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c79d29-eee5-4399-bb87-6e2b40a2b2ba None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.141781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c79d29-eee5-4399-bb87-6e2b40a2b2ba None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.141781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 351/697] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:06.143346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c79d29-eee5-4399-bb87-6e2b40a2b2ba None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.163317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365de812-72e6-4eb4-ba7a-4fa3a89a9209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.163968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365de812-72e6-4eb4-ba7a-4fa3a89a9209 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.164222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365de812-72e6-4eb4-ba7a-4fa3a89a9209 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.164487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365de812-72e6-4eb4-ba7a-4fa3a89a9209 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.175441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365de812-72e6-4eb4-ba7a-4fa3a89a9209 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.177141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365de812-72e6-4eb4-ba7a-4fa3a89a9209 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-365de812-72e6-4eb4-ba7a-4fa3a89a9209 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:06.192424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:06.193464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:06.194431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:06.194431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:06.194431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:06.194431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:06.194431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 347/698] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:06.194721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1657d00-4156-4bce-902c-2c7f0c08e739 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.195454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1657d00-4156-4bce-902c-2c7f0c08e739 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.195551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1657d00-4156-4bce-902c-2c7f0c08e739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.215658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b1657d00-4156-4bce-902c-2c7f0c08e739 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:06.216790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b1657d00-4156-4bce-902c-2c7f0c08e739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:06.234192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:06.262893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c79d29-eee5-4399-bb87-6e2b40a2b2ba None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.293598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 348/699] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:06.294725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ccb99-38ec-4b90-82f5-c898412221b7 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.295054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ccb99-38ec-4b90-82f5-c898412221b7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.295335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2ccb99-38ec-4b90-82f5-c898412221b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.315121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-cf2ccb99-38ec-4b90-82f5-c898412221b7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:06.316357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cf2ccb99-38ec-4b90-82f5-c898412221b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:06.318879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.320802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.321035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.321308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.322018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:06.322370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.323781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.342467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 352/700] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:06.343446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.398418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:06.462160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0ea190-fb02-4990-88ae-080ea46f71e8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.502332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.503199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.503511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.503834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.504613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:06.504984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.507245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.514437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 349/701] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:06.515510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2ccb99-38ec-4b90-82f5-c898412221b7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.534978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 353/702] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:06.534978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.624652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:06.639482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b35f0c-838d-4978-9c89-fa065a0d6b92 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.648786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:06.652742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2ccb99-38ec-4b90-82f5-c898412221b7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.677557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.678209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.678480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.678750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.686457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.687497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.687934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.688186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.688488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.688760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.699648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:06.699906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:06.700820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:06.707566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:06.707984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.708065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 354/703] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:06.708065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:06.725489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:06.726495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:06.727426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:06.727426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:06.727426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:06.727426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:06.727426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 350/704] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:06.727710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.819542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02e5a52-929d-4c8e-8b28-beb95cf0793d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.826275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738caeb5-b250-4111-ad00-405921f5ca14 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:06.855981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.856904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.857533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:06.858103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.858609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.864466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:06.865122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:06.865389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:06.865671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:06.866039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.866304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:06.867527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.867691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:06.879304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:06.880193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:06.890541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:06.891006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:06.891914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:06.892757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:06.893535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 355/706] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:06.893612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 351/706] 10.222.0.42 () {56 vars in 1129 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:06.893680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.893748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:06.922910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:07.001131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643c6dba-1ccd-44c5-a4ce-a2ce1bfed4e2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.002795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be6d25-e901-49f9-a708-1662fc6ef112 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.039936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.040568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.040850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.041128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.041831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:07.042160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.042817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.043497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.043605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.043780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.044056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.044820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:07.045129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.046555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.063995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 356/707] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:07.063995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.064246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 352/708] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:07.064246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.118907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:07.155706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:07.167047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316960ca-d3bd-46bf-8b83-790509583bc0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.169155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af170b20-cf9d-4a7f-b428-129a0e846bab None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.205064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.205690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.205970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.206236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.206921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:07.207122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.207297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.207746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.208077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.208351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.208645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.209111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:07.209415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.210865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.214224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 353/709] 10.222.0.42 () {56 vars in 1124 bytes} [Tue Jul 26 02:58:07 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:07.215297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.223981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 357/710] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:07.224850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.330300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af5c0f-43e1-411c-9b8e-a348b3d43a4a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.347142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6be7352-4f93-43cf-99aa-81b5f5637df8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.365070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.365692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.365967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.366229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.366704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:07.367047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=afebddca912546119eea61f568bbd60d) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:07.367369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.368809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.374790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 358/711] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:07 2022] GET /identity/v3/role_assignments?user.id=afebddca912546119eea61f568bbd60d&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:07.375603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.386104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.387348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.387630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.387898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.388948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.390768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.391296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:07.418772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 354/712] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:58:07 2022] POST /identity/v3/users => generated 254 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:07.419581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.501819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb10e866-aa71-444d-8692-ca60fe1bc4a2 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.557382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d74b1ab-743b-4342-b811-3a1be6f91559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.559780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d74b1ab-743b-4342-b811-3a1be6f91559 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.560051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d74b1ab-743b-4342-b811-3a1be6f91559 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.560358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d74b1ab-743b-4342-b811-3a1be6f91559 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.561080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d74b1ab-743b-4342-b811-3a1be6f91559 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:07.561417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d74b1ab-743b-4342-b811-3a1be6f91559 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.562813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d74b1ab-743b-4342-b811-3a1be6f91559 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.571824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 359/713] 10.222.0.42 () {56 vars in 1124 bytes} [Tue Jul 26 02:58:07 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:07.573313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa5dae-a92f-474f-8504-0e11cc12f2f4 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.573603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa5dae-a92f-474f-8504-0e11cc12f2f4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.573882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa5dae-a92f-474f-8504-0e11cc12f2f4 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.575714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 360/714] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:07.581537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db94b3-4641-49c2-a731-9e4ab77aa975 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.581873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db94b3-4641-49c2-a731-9e4ab77aa975 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.582115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95db94b3-4641-49c2-a731-9e4ab77aa975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.600032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ea0ad9-156e-4bbf-921b-9ecc9e8d8042 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.608093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-95db94b3-4641-49c2-a731-9e4ab77aa975 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:07.609665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-95db94b3-4641-49c2-a731-9e4ab77aa975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:07.653592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.654796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.655243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.655541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.656387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=e2c09978fa3f44d895cbc17efdca2b32) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:07.656698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.658509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.666039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 355/715] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:07 2022] GET /identity/v3/role_assignments?user.id=e2c09978fa3f44d895cbc17efdca2b32&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:07.706699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 361/716] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:07.731080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.789072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:07.863623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecdf484-8e70-4014-bac2-b9eb5359e11a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:07.903126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:07.903790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:07.904066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:07.904337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] PATH_INFO: `/v3/services/network` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:07.904976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:07.906363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:07.916906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:07.918054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:07.919180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 356/717] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:07 2022] GET /identity/v3/services/network => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:07.924288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db94b3-4641-49c2-a731-9e4ab77aa975 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:07.939331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:08.098243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95db94b3-4641-49c2-a731-9e4ab77aa975 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:08.140833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9757cb-4b95-4599-86b7-b288185d8579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:08.141465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9757cb-4b95-4599-86b7-b288185d8579 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:08.141742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9757cb-4b95-4599-86b7-b288185d8579 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:08.142012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9757cb-4b95-4599-86b7-b288185d8579 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:08.142694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9757cb-4b95-4599-86b7-b288185d8579 None admin] RBAC: Adding query filter params (name=network) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:08.143017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9757cb-4b95-4599-86b7-b288185d8579 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:08.144392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9757cb-4b95-4599-86b7-b288185d8579 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:08.149395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 362/718] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:07 2022] GET /identity/v3/services?name=network => generated 124 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:08.154668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:08.262662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c32afa-ed5f-41c0-a1f2-375dea866cfc None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:08.264318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:08.298376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647518cf-b218-4594-9be5-3d04333b2393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:08.299032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647518cf-b218-4594-9be5-3d04333b2393 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:08.299315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647518cf-b218-4594-9be5-3d04333b2393 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:08.299582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647518cf-b218-4594-9be5-3d04333b2393 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:08.300265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647518cf-b218-4594-9be5-3d04333b2393 None admin] RBAC: Adding query filter params (type=network) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:08.300569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647518cf-b218-4594-9be5-3d04333b2393 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:08.301951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647518cf-b218-4594-9be5-3d04333b2393 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:08.308215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 357/719] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:08 2022] GET /identity/v3/services?type=network => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:09.938584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad57e32-c362-4ee7-bb76-00ded377246f None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:09.939159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad57e32-c362-4ee7-bb76-00ded377246f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:09.939159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad57e32-c362-4ee7-bb76-00ded377246f None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:09.940757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 363/720] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:09.945464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9721126f-2b80-4ff2-a627-a66ded0b80f6 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:09.945742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9721126f-2b80-4ff2-a627-a66ded0b80f6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:09.946017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9721126f-2b80-4ff2-a627-a66ded0b80f6 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:09.947540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 364/721] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:09.949049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55fc3f6-276a-4f05-883e-ea6febf202c6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:09.949473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55fc3f6-276a-4f05-883e-ea6febf202c6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:09.949570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55fc3f6-276a-4f05-883e-ea6febf202c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:09.959851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d1d72-e4af-4bd5-b2bb-f0a56f5e4910 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:09.960140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d1d72-e4af-4bd5-b2bb-f0a56f5e4910 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:09.960421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749d1d72-e4af-4bd5-b2bb-f0a56f5e4910 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:09.972560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c55fc3f6-276a-4f05-883e-ea6febf202c6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:09.973751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c55fc3f6-276a-4f05-883e-ea6febf202c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:09.996198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-749d1d72-e4af-4bd5-b2bb-f0a56f5e4910 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:09.997428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-749d1d72-e4af-4bd5-b2bb-f0a56f5e4910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:10.105097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 358/722] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:10.117393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6b026-c44b-4228-afd2-e15651ef4b57 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.117672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6b026-c44b-4228-afd2-e15651ef4b57 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.117942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f6b026-c44b-4228-afd2-e15651ef4b57 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.119642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 359/723] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:58:10.125094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71568c0-bdea-44ec-9140-ef9f26f2bdf5 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.125379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71568c0-bdea-44ec-9140-ef9f26f2bdf5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.125655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71568c0-bdea-44ec-9140-ef9f26f2bdf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.127054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:10.133436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 365/724] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:10.134297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749d1d72-e4af-4bd5-b2bb-f0a56f5e4910 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.149445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c71568c0-bdea-44ec-9140-ef9f26f2bdf5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:10.150626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c71568c0-bdea-44ec-9140-ef9f26f2bdf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:10.234794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 360/725] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 26 02:58:10.235626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71568c0-bdea-44ec-9140-ef9f26f2bdf5 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.287171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:10.289186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749d1d72-e4af-4bd5-b2bb-f0a56f5e4910 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:10.326818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:10.327500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.327780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.328195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.338532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:10.343502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:10.344886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 366/726] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:10.345861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.373096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71568c0-bdea-44ec-9140-ef9f26f2bdf5 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:10.409114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:10.413534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be4dcd0-86d2-4d9a-9da7-d72e7f68af91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:10.414394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be4dcd0-86d2-4d9a-9da7-d72e7f68af91 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.414674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be4dcd0-86d2-4d9a-9da7-d72e7f68af91 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.414957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be4dcd0-86d2-4d9a-9da7-d72e7f68af91 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.426046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be4dcd0-86d2-4d9a-9da7-d72e7f68af91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-7be4dcd0-86d2-4d9a-9da7-d72e7f68af91 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:10.431163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:10.432271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:10.433144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:10.435469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 361/727] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:10.438132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a33ffa-adef-462b-b6fc-71623764e2fc None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.438420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a33ffa-adef-462b-b6fc-71623764e2fc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.438692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a33ffa-adef-462b-b6fc-71623764e2fc None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.440361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 362/728] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:10.445243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a33ffa-adef-462b-b6fc-71623764e2fc None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.548571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:10.567401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a33ffa-adef-462b-b6fc-71623764e2fc None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:10.578575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc0b98d-c869-4680-a03b-49ca6fee9a6d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:10.601978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:10.602612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.602938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.603237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.603978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:10.604287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:10.605798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:10.615042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 363/729] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:10.615909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.621458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b927dd-2aee-4df0-8953-874a4f1b279d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:10.622095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b927dd-2aee-4df0-8953-874a4f1b279d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.622371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b927dd-2aee-4df0-8953-874a4f1b279d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.622642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b927dd-2aee-4df0-8953-874a4f1b279d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.634572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b927dd-2aee-4df0-8953-874a4f1b279d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-c7b927dd-2aee-4df0-8953-874a4f1b279d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:10.640215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:10.641158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:10.641995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 367/730] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:10.643355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16e47a2-b89c-4a32-85e2-5b92a9b9a22f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.643636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16e47a2-b89c-4a32-85e2-5b92a9b9a22f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.643904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16e47a2-b89c-4a32-85e2-5b92a9b9a22f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.662335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d16e47a2-b89c-4a32-85e2-5b92a9b9a22f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:10.663518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d16e47a2-b89c-4a32-85e2-5b92a9b9a22f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:10.728484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af407e2-cb2b-41fb-9b33-1c80850cc942 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:10.735863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 368/731] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:10.743838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16e47a2-b89c-4a32-85e2-5b92a9b9a22f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.763440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:10.764070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.764347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.764626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.765344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:10.765646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:10.767039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:10.775796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 364/732] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:10.776726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.792124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:10.805626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:10.858147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16e47a2-b89c-4a32-85e2-5b92a9b9a22f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:10.894056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:10.894712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.895003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.895277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.897282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee2583-9cda-4ac6-bf35-892c140b2f24 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:10.905907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:10.907441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:10.922714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:10.923708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:10.924588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:10.924588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:10.924588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:10.924588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:10.924588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 369/733] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:10.924868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:10.940691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:10.941317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:10.941592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:10.941863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:10.942576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:10.942875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:10.944272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:10.953500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 365/734] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:10.954377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.025684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3430f1-1cac-48e4-b496-efd3645cce05 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.027412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.028577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.056900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299280bd-b612-469b-911b-0325f67866b5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.060242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc340c6c-92dc-4728-b6bb-4c086ba19e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.060871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc340c6c-92dc-4728-b6bb-4c086ba19e25 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.061144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc340c6c-92dc-4728-b6bb-4c086ba19e25 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.061419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc340c6c-92dc-4728-b6bb-4c086ba19e25 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.062113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc340c6c-92dc-4728-b6bb-4c086ba19e25 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.063526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc340c6c-92dc-4728-b6bb-4c086ba19e25 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.080227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 370/735] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:10 2022] POST /identity/v3/services => generated 251 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:11.081564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca41470-b1de-4929-adf3-7ada3b45b43a None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.081932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca41470-b1de-4929-adf3-7ada3b45b43a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.082121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca41470-b1de-4929-adf3-7ada3b45b43a None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.083412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 371/736] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:11.088667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca41470-b1de-4929-adf3-7ada3b45b43a None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.098895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.099624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.099904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.100184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.114849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.116502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:11.137237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:11.138281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:11.139687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:11.139687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:11.139687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:11.139687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:11.139687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 366/737] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:11.140004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.217484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca41470-b1de-4929-adf3-7ada3b45b43a None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.262444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f8051-9a2c-4fcf-a7f5-2cc11fef374d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.263438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29ac3d8-ad35-48ef-ba5a-03ef932cac8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.264174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29ac3d8-ad35-48ef-ba5a-03ef932cac8a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.264458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29ac3d8-ad35-48ef-ba5a-03ef932cac8a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.264737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29ac3d8-ad35-48ef-ba5a-03ef932cac8a None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.265451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29ac3d8-ad35-48ef-ba5a-03ef932cac8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:11.266606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29ac3d8-ad35-48ef-ba5a-03ef932cac8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.268069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29ac3d8-ad35-48ef-ba5a-03ef932cac8a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.280176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 372/738] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:11.281532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf92a4b-a2fe-4d09-b053-267995734a1d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.281821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf92a4b-a2fe-4d09-b053-267995734a1d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.282097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf92a4b-a2fe-4d09-b053-267995734a1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.289503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.302985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-eaf92a4b-a2fe-4d09-b053-267995734a1d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:11.304173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-eaf92a4b-a2fe-4d09-b053-267995734a1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:11.304738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.305395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.305673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.305939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.317861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.319407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.321743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:11.334140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:11.335086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:11.335975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:11.335975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:11.335975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:11.335975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:11.339260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 367/739] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:11.340814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.381111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 373/740] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:11 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 26 02:58:11.381821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf92a4b-a2fe-4d09-b053-267995734a1d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.457736 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abe57e1-4190-4dae-acc9-9fc65930e526 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.487828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf92a4b-a2fe-4d09-b053-267995734a1d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.494606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.495260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.495535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.495836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.496636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:11.496937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.498319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.518984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 368/741] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:11.518984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.537154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.537802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.538095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.538354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.546614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.548126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.551728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:11.562750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:11.563769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 374/742] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:11.571352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.586240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.626443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77036a8-a720-419e-9358-9f826c0954ef None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.667237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.667934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.668213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.668488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.669192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:11.669499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.670890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.679996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055a57ed-f195-4fc7-b2e8-bfad5018f680 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.684446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 369/743] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:11.685269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.718755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.719878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.720230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.720571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.721345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.722785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.738841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 375/744] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:11 2022] POST /identity/v3/services => generated 240 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:11.739597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.802632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.816623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4dbdd67-f448-43a7-9811-c1eaae9e875a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.821282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:11.871702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.872632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.873051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.873497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.877345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e81dea4-7ce8-4532-83c3-d14f8aeb7f95 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:11.883378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.885108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:11.899192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:11.900278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:11.901441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:11.908334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 370/745] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/users/swift => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:11.909600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:11.918262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:11.919221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:11.919653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:11.920045 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:11.921870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:11.922327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:11.923933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:11.944253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 376/746] 10.222.0.42 () {56 vars in 1149 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:11.945399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.032599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdeec12-55db-4fac-83a8-f61e6bc4ab04 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.054717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620387ba-5dab-4831-9919-bd22b994c372 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.066676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.067365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.067648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.067926 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] PATH_INFO: `/v3/users/nova` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.075858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.077331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.082892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:12.090870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:12.092152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:12.093326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:12.096790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 371/747] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:12.097757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.105602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.106371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.106752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.107144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.108058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:12.108457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.110606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.111892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:12.139106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 377/748] 10.222.0.42 () {56 vars in 1135 bytes} [Tue Jul 26 02:58:11 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:12.140205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.218142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6522d-7d0c-4532-aa25-0bf7a64d04da None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.244970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9160f170-6d44-46fb-bb75-6fc7f03ed631 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.258567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.259448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.259821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.260197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.285525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.286693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.287187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.288038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.289279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:12.289729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.292194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.316109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 378/749] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:12.317196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.338876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.340554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.352090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:12.377612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:12.449294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b5bcbb-b9db-48d3-a38b-5b166417db6e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.459128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 372/750] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:12.460371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.490433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.491152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.491409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.491694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.576818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767b3b0a-9790-4bf3-b22b-b427a6af0fee None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.581184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.582875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.587534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:12.621963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.623042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.623520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.623986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.628577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:12.640855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.643251 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:12.662330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:12.663297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:12.664217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:12.664217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:12.664217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:12.664217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:12.666250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 373/751] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:12.667206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.741442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 379/752] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:12.743201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.786094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab20965-401d-4f6a-ba89-11beaf166f03 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.826783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.828282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.828747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.829159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.843587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.845888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:12.864137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:12.865193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:12.866198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:12.866198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:12.866198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:12.866198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:12.868037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 374/753] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:12.868962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.885858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:12.917753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc466847-c68c-48b0-96f0-8e8be28f92c2 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:12.960250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:12.961632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:12.962260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:12.962807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:12.963710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:12.964159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:12.966471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:12.981578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 380/754] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:12.982627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:12.986854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:13.046888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e0a998-3ee6-45c0-8d57-5151db32c896 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.089230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.090218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.090642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.091105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.105763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.107841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:13.125119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:13.126162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:13.127229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:13.127229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:13.127229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:13.127229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:13.129617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 375/755] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:13.130600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.140946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf22b70f-d7c4-4c26-83bb-344fba95c272 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.198303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.199646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.200247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.200830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.202239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:13.202931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.204976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:13.206282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.235329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 381/756] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:13.237300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.249701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91416bc5-e5d2-4a64-b25c-fed04bc40395 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.289004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.289915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.290337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.290990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.299896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.301557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:13.315147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:13.316177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:13.317205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:13.319276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 376/757] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:13.320227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.363504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:13.431382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d7e1a1-559e-4abc-9825-b577202d0f5a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.448973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333a7e95-8b5c-4c60-95bd-477c5628f0e1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.479906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:13.484203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.485031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.485310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.485611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.486323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:13.486639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.488050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.493727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a74236-6e7f-4639-bc10-40e0336bbe6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.494614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a74236-6e7f-4639-bc10-40e0336bbe6a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.495576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a74236-6e7f-4639-bc10-40e0336bbe6a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.496174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a74236-6e7f-4639-bc10-40e0336bbe6a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.500304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 382/758] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:13.500965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.505907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a74236-6e7f-4639-bc10-40e0336bbe6a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.507411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a74236-6e7f-4639-bc10-40e0336bbe6a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-37a74236-6e7f-4639-bc10-40e0336bbe6a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:13.519563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:13.520480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:13.521365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 377/759] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:13.522400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa6baad-af98-4e42-943f-e613392ab3b2 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.522400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa6baad-af98-4e42-943f-e613392ab3b2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.522400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa6baad-af98-4e42-943f-e613392ab3b2 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.523767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 378/760] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:13 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:13.524302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa6baad-af98-4e42-943f-e613392ab3b2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.629527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c2830-8fdf-49b7-aaec-57cf69306861 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.631517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa6baad-af98-4e42-943f-e613392ab3b2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.655198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:13.668016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.668638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.668916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.669026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c166dffe-6fb9-4ac9-8540-3179bfb8d617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.669239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.669623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166dffe-6fb9-4ac9-8540-3179bfb8d617 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.669905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166dffe-6fb9-4ac9-8540-3179bfb8d617 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.670006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:13.670175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166dffe-6fb9-4ac9-8540-3179bfb8d617 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.670268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.671874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.678165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c166dffe-6fb9-4ac9-8540-3179bfb8d617 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.680006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c166dffe-6fb9-4ac9-8540-3179bfb8d617 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.689701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 383/761] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:13.689701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-c166dffe-6fb9-4ac9-8540-3179bfb8d617 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:13.695773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:13.696706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:13.697592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 379/762] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:13.698513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6949a68-4543-4957-bf20-672ed38873d0 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.698513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6949a68-4543-4957-bf20-672ed38873d0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.699212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6949a68-4543-4957-bf20-672ed38873d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.718554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a6949a68-4543-4957-bf20-672ed38873d0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:13.720247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a6949a68-4543-4957-bf20-672ed38873d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:13.765771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:13.795906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d38a90e-1ae8-4b4a-b274-7580d1a5f003 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.797787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 380/763] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:13 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:13.798757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6949a68-4543-4957-bf20-672ed38873d0 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.839370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.840068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.840352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.840660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.841436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:13.841740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.843221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.858207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 384/764] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:13.859313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.902277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6949a68-4543-4957-bf20-672ed38873d0 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:13.906488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:13.941302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08679ec-0744-41ec-858b-01945c15c219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:13.941979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08679ec-0744-41ec-858b-01945c15c219 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.942253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08679ec-0744-41ec-858b-01945c15c219 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.942531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08679ec-0744-41ec-858b-01945c15c219 None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/e2c09978fa3f44d895cbc17efdca2b32/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.944551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08679ec-0744-41ec-858b-01945c15c219 None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=e2c09978fa3f44d895cbc17efdca2b32, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:13.946021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08679ec-0744-41ec-858b-01945c15c219 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:13.960643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 381/765] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:58:13 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/e2c09978fa3f44d895cbc17efdca2b32/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:58:13.961867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c176fc-fe39-413b-9f49-b942ba83fec0 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:13.962151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c176fc-fe39-413b-9f49-b942ba83fec0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:13.962428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c176fc-fe39-413b-9f49-b942ba83fec0 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:13.963711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 382/766] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:13.964425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c176fc-fe39-413b-9f49-b942ba83fec0 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:13.974621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fda41f1-9ed1-4f19-9429-617e937c23ee None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.007097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:14.025121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.025771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.026050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.026324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.027161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:14.027476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.028938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.047353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 385/767] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:14.047353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.100199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c176fc-fe39-413b-9f49-b942ba83fec0 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.143341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b431c48c-d1d5-48c6-a320-fa77bb4c327d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.144218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431c48c-d1d5-48c6-a320-fa77bb4c327d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.144330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431c48c-d1d5-48c6-a320-fa77bb4c327d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.144529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b431c48c-d1d5-48c6-a320-fa77bb4c327d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.145165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431c48c-d1d5-48c6-a320-fa77bb4c327d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.146546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b431c48c-d1d5-48c6-a320-fa77bb4c327d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-b431c48c-d1d5-48c6-a320-fa77bb4c327d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:14.156068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:14.156988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:14.160377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 383/768] 10.222.0.42 () {56 vars in 1129 bytes} [Tue Jul 26 02:58:13 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:14.161218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a148ce3-26aa-4b00-bc60-ca49aad45222 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.161218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a148ce3-26aa-4b00-bc60-ca49aad45222 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.161218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a148ce3-26aa-4b00-bc60-ca49aad45222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.174498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07770f9-e7d1-47b1-823a-c5aca3350f5c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.176247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:14.179530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-5a148ce3-26aa-4b00-bc60-ca49aad45222 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:14.180779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5a148ce3-26aa-4b00-bc60-ca49aad45222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:14.228751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.229740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.230059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.230458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.231590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=b211db19066a4767b09f1fcae194085a) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:14.231975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.234017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.241130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 386/769] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/role_assignments?user.id=b211db19066a4767b09f1fcae194085a&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:14.242642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.267879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 384/770] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:14.269147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a148ce3-26aa-4b00-bc60-ca49aad45222 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.300343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:14.386618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a148ce3-26aa-4b00-bc60-ca49aad45222 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.390329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25909c73-17d1-4e28-a548-53d21d2991e8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.427117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.428072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.428506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.428918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.429939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:14.430413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.431544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.432030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.432406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.432784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.433048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.433900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=afebddca912546119eea61f568bbd60d) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:14.434332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.436179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.443225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 387/772] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/role_assignments?user.id=afebddca912546119eea61f568bbd60d&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:14.443485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 385/772] 10.222.0.42 () {56 vars in 1124 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:14.444590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.445937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.488488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:14.595804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cedbe21-0c06-4821-b841-0f59e7370bbb None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.599310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:14.605221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2e41d7-c850-4ce3-955a-70e4f3cd0526 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.668118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.668785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.669060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.669331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] PATH_INFO: `/v3/services/network` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.669972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.671448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.672978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.674360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.674844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.675325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.676661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:14.677514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.679352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:14.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:14.685801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:14.686840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 388/773] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/services/network => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:14.692539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.692825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 386/774] 10.222.0.42 () {56 vars in 1124 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:14.695363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.830720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:14.838914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08e27ef-f489-44ed-a935-af904dc67130 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.877224 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.878115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.878697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.879142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.880020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] RBAC: Adding query filter params (name=network) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:14.880467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.882105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:14.887944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 389/775] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/services?name=network => generated 124 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:14.896118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:14.910729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2db4e67-9670-4ff7-9296-6b03478cb689 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:14.978696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:14.979614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:14.980053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:14.980527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:14.981503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] RBAC: Adding query filter params (interface=public, service_id=97edc2be7e944722b21742b66130e6ac, region_id=RegionOne) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:14.981961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:14.986205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:15.000958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 387/776] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/endpoints?service_id=97edc2be7e944722b21742b66130e6ac&interface=public®ion_id=RegionOne => generated 194 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:15.008812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:15.055837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:15.097823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e09fec-8d90-4a0d-9b87-9c07f6b13b89 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:15.162576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:15.163515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:15.163937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:15.164348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:15.165177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] RBAC: Adding query filter params (type=network) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:15.165650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:15.167193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:15.172901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 390/777] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:14 2022] GET /identity/v3/services?type=network => generated 350 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:15.179004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:15.206525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:15.225797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a2b121-6f4a-4e55-b775-f454d9caa226 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:15.290614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c465ea7f-47f9-4c8c-8f21-59f68d84fdf0 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:15.326754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:15.327683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:15.328101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:15.328546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:15.329463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] RBAC: Adding query filter params (interface=public, service_id=f19b2224be2a45a385e8578dd19ffecd, region_id=RegionOne) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:15.329919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:15.331475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:15.339903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 391/778] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 02:58:15 2022] GET /identity/v3/endpoints?service_id=f19b2224be2a45a385e8578dd19ffecd&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:15.345769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:15.367667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e4eb30-9205-4285-b744-a0ae5b155d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:15.368321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4eb30-9205-4285-b744-a0ae5b155d15 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:15.368610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4eb30-9205-4285-b744-a0ae5b155d15 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:15.369148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e4eb30-9205-4285-b744-a0ae5b155d15 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:15.369558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e4eb30-9205-4285-b744-a0ae5b155d15 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:15.370988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e4eb30-9205-4285-b744-a0ae5b155d15 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:15.389614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 388/779] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:15 2022] GET /identity/v3/services => generated 772 bytes in 382 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:15.439327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:15.452497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59cd048-6535-4694-bd66-c34e7338ea99 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:15.489415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa3bfc8-5746-44d3-ab77-b7002b05de20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:15.490040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3bfc8-5746-44d3-ab77-b7002b05de20 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:15.490319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3bfc8-5746-44d3-ab77-b7002b05de20 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:15.490592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3bfc8-5746-44d3-ab77-b7002b05de20 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:15.491291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa3bfc8-5746-44d3-ab77-b7002b05de20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:15.492672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa3bfc8-5746-44d3-ab77-b7002b05de20 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:15.498324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 392/780] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:15 2022] GET /identity/v3/services => generated 772 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:16.344318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8a42a1-15d6-4eed-97bc-8586efdc03fe None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.345328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8a42a1-15d6-4eed-97bc-8586efdc03fe None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.345811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8a42a1-15d6-4eed-97bc-8586efdc03fe None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.347710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 389/781] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:16.353591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8490f36-6d36-41b2-ab4d-2d533160c3da None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.354469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8490f36-6d36-41b2-ab4d-2d533160c3da None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.354976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8490f36-6d36-41b2-ab4d-2d533160c3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.380032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c8490f36-6d36-41b2-ab4d-2d533160c3da None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:16.381245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c8490f36-6d36-41b2-ab4d-2d533160c3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:16.469957 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 393/782] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:16.505270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8a42a1-15d6-4eed-97bc-8586efdc03fe None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:16.627606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0661ce-3478-43e6-916d-6018431661ae None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.627949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0661ce-3478-43e6-916d-6018431661ae None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.628172 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0661ce-3478-43e6-916d-6018431661ae None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.629723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 394/783] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:16.634733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5496b5c4-bdd1-4f69-9f0b-e83194bfaf54 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.635043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5496b5c4-bdd1-4f69-9f0b-e83194bfaf54 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.635335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5496b5c4-bdd1-4f69-9f0b-e83194bfaf54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.645534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:16.664120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8a42a1-15d6-4eed-97bc-8586efdc03fe None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:16.680118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5496b5c4-bdd1-4f69-9f0b-e83194bfaf54 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:16.681351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5496b5c4-bdd1-4f69-9f0b-e83194bfaf54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:16.701420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:16.702083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.702366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.702641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.722659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:16.727614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:16.728532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:16.729410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 390/784] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:16.735959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:16.809946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 395/785] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:16.838376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f161db-cd07-4200-8d56-e94ae8a462f3 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.838663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f161db-cd07-4200-8d56-e94ae8a462f3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.838992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f161db-cd07-4200-8d56-e94ae8a462f3 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.840663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 396/786] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:58:16.844427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f161db-cd07-4200-8d56-e94ae8a462f3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:16.860374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:16.879987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d6f66b-6dfb-4b7c-9dd3-9364268a6a83 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:16.923424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc475a9d-52ca-4891-a5c6-c43949994ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:16.924234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc475a9d-52ca-4891-a5c6-c43949994ab6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.924638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc475a9d-52ca-4891-a5c6-c43949994ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.925010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc475a9d-52ca-4891-a5c6-c43949994ab6 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.926103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc475a9d-52ca-4891-a5c6-c43949994ab6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:16.926533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc475a9d-52ca-4891-a5c6-c43949994ab6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:16.929243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc475a9d-52ca-4891-a5c6-c43949994ab6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:16.942093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 391/787] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:16.944158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62892ae6-6a5a-4a19-97aa-44d97160a3c6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:16.944446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62892ae6-6a5a-4a19-97aa-44d97160a3c6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:16.944720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62892ae6-6a5a-4a19-97aa-44d97160a3c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:16.963459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:16.970496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-62892ae6-6a5a-4a19-97aa-44d97160a3c6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:16.972557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-62892ae6-6a5a-4a19-97aa-44d97160a3c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:16.973798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f161db-cd07-4200-8d56-e94ae8a462f3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.030117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.031344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.031840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.032374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.044836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:17.050826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:17.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:17.053023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:17.055752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 392/788] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:17.057020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62892ae6-6a5a-4a19-97aa-44d97160a3c6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.057708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 397/789] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:58:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:17.063444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.141904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:17.176968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62892ae6-6a5a-4a19-97aa-44d97160a3c6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.179515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4557e42a-cb7e-4055-9a31-69fdbac947a4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.212867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.213517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.213797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.214072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.216855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:17.218799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.220073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.220639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.221111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.222361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:17.223844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.225331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.225697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.226793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.236686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 398/790] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:17.238177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:17.243132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:17.244085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:17.245552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:17.245552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:17.245552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:17.245552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:17.245552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 393/791] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:17.245552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.354108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b800d1-f84a-453c-8f91-1b287abc193e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.354645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c41b602-588d-41b1-88cc-49d66b10055d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.387723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:17.393949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.394440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.395091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.395306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.395419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.395680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.396018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.396613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.408352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.408920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:17.409832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:17.415204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:17.416361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:17.417394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:17.419881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 399/792] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:17.420974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:17.434245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:17.435420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:17.436535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:17.436535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:17.436535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:17.436535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:17.436535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 394/793] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:17.437010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.482554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:17.549470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9501a799-56f9-4c54-baa6-241bfe52b5db None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.553926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44be3a20-4835-4715-8420-9c52afd09e04 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.585674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.586301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.586589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.586865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.587627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:17.587938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.589463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.598831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.599816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.600275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.600744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.601645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:17.602128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.603309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 395/794] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:17.604066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.604221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.620279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 400/795] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:17.621992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.684573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:17.739167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8efa670-a203-468b-a266-5fddc0db15ae None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.739949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa317b9-f573-42f2-b8ea-343f45cdb5bb None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.742922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:17.779859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.780944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.781404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.781855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.784304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.784940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.785224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.785915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.786636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:17.786968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.788354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.799991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.801475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 396/796] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:17.801586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.802696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:17.820593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:17.821569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:17.822488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:17.822488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:17.822488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:17.822488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:17.829617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 401/797] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:17.830936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.913144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5121e41-f73b-4ebb-9aac-f775ee5116cb None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.933832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9091f6f3-b286-467e-ab47-3c0e3cad7b29 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:17.936216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:17.948179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c98491-e22b-467b-b663-114c158780ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.948857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c98491-e22b-467b-b663-114c158780ea None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.949161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c98491-e22b-467b-b663-114c158780ea None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.949435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c98491-e22b-467b-b663-114c158780ea None admin] PATH_INFO: `/v3/users/nova` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.956610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c98491-e22b-467b-b663-114c158780ea None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.958067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c98491-e22b-467b-b663-114c158780ea None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.969539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75e3b4e-3f9a-4352-a429-e3ba4e8ee0d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-71c98491-e22b-467b-b663-114c158780ea None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:17.969976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:17.970936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:17.971931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:17.972464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e3b4e-3f9a-4352-a429-e3ba4e8ee0d7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:17.972464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e3b4e-3f9a-4352-a429-e3ba4e8ee0d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:17.972464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e3b4e-3f9a-4352-a429-e3ba4e8ee0d7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:17.972654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 397/798] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:17.972654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c98491-e22b-467b-b663-114c158780ea None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:17.982179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75e3b4e-3f9a-4352-a429-e3ba4e8ee0d7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:17.983663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75e3b4e-3f9a-4352-a429-e3ba4e8ee0d7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-b75e3b4e-3f9a-4352-a429-e3ba4e8ee0d7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:17.994771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:17.995797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:17.996897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:17.997718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.000942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 402/799] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:18.002109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13c3d26-c8e7-4965-9bd0-779ac6367107 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.002398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13c3d26-c8e7-4965-9bd0-779ac6367107 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.002676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13c3d26-c8e7-4965-9bd0-779ac6367107 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.003982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 403/800] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:18.004986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731fa840-aec1-4473-b9cd-38ad5df1e541 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.005354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731fa840-aec1-4473-b9cd-38ad5df1e541 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.005630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731fa840-aec1-4473-b9cd-38ad5df1e541 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.006993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 404/801] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:18 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:18.011415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731fa840-aec1-4473-b9cd-38ad5df1e541 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.079803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c98491-e22b-467b-b663-114c158780ea None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.114885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.115530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.115674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731fa840-aec1-4473-b9cd-38ad5df1e541 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.115809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.116085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.116776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:18.117072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.118456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.130210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 398/802] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:18.131029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.151557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd4d211-6970-43d3-a01f-d87ab203433b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.152269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd4d211-6970-43d3-a01f-d87ab203433b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.152551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd4d211-6970-43d3-a01f-d87ab203433b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.152829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd4d211-6970-43d3-a01f-d87ab203433b None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.153637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd4d211-6970-43d3-a01f-d87ab203433b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:18.153950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd4d211-6970-43d3-a01f-d87ab203433b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.155433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd4d211-6970-43d3-a01f-d87ab203433b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.173542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 405/803] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:18.174916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0d5129-6050-476e-8d21-0dbd7476c940 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.175240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0d5129-6050-476e-8d21-0dbd7476c940 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.175514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0d5129-6050-476e-8d21-0dbd7476c940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.192984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-7b0d5129-6050-476e-8d21-0dbd7476c940 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:18.194103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7b0d5129-6050-476e-8d21-0dbd7476c940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:18.211009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.217832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.237613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0b15e-f9cb-4246-ab05-19bcdfe08818 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.263721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 406/804] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:18.264993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a47530-6272-43b8-a1f4-290750880823 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.265295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a47530-6272-43b8-a1f4-290750880823 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.265564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a47530-6272-43b8-a1f4-290750880823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.280195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.280794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.281072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.281341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.282141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:18.282447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.283902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.290847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-29a47530-6272-43b8-a1f4-290750880823 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:18.292072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-29a47530-6272-43b8-a1f4-290750880823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:18.301125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 399/805] 10.222.0.42 () {56 vars in 1149 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:18.301984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.358091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 407/806] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:18.358970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a47530-6272-43b8-a1f4-290750880823 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.410315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c79bab0-9d41-47e3-8c6f-0cfba5de0ddf None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.443194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.443763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.444036 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.444345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] PATH_INFO: `/v3/users/swift` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.444704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.451905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.453356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.455941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.463788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a47530-6272-43b8-a1f4-290750880823 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:18.465688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:18.466640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 400/807] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:18.467920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.499813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.500434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.500720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.500970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.565511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a87036-cf43-4332-92f9-f272b2dc67cb None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.567242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.568701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.599169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.599835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.600297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.600778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] PATH_INFO: `/v3/services/network` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.601433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.602820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:18.611755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:18.612733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:18.613583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 401/808] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/services/network => generated 88 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:18.614380 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.654546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.675514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 408/809] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:18.675717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.676338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.721172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d2e96c-7281-4721-8ec9-2deccebc41f8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.753399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.754047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.754327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.754598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.765142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.766608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.774930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d17ffa-e5f3-46cc-8acc-590a792cc848 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:18.780642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:18.781557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:18.782527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:18.782527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:18.782527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:18.782527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:18.782527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 402/810] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:18.783072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.815142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.815793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.816070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.816343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.816954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.818324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:18.827090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:18.828184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:18.829026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 409/811] 10.222.0.42 () {56 vars in 1129 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:18.829855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.874633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.885065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3aac309-6526-4d4f-98cc-74a6608a3832 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.900903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:18.919457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.920080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.920358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.920629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.921433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:18.921739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.923207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.928019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad44c12-8f46-47ea-8c2d-16f107487d2d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:18.940066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 403/812] 10.222.0.42 () {56 vars in 1135 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:18.940875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:18.963009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:18.963881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:18.964167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:18.964453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:18.965219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] RBAC: Adding query filter params (name=network) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:18.965531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:18.966915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:18.972696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 410/813] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:18.973338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.044928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1f501-59ee-4fbb-b44b-da7774460123 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.071670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122e11fb-ac66-4979-bfd4-93b4362c5469 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.078518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.079196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.079468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.079736 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.082216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.090139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.091642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.106016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.106637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:19.106744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:19.107855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:19.108733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:19.108733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:19.108733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:19.108733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:19.108964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.108964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.109086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 404/814] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:19.109162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:19.109162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.109283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.109804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.121657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 411/815] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:19.122438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.138884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.213394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b078c4-4555-4e13-80c6-292128d5123e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.221138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2e868-d890-4890-b8ef-5bfd51576dd8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.247225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.247866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.248151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.248482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.249168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:19.249475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.250865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.255605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 405/816] 10.222.0.42 () {56 vars in 1124 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:19.256092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.256374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.256790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.257082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.257350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.267752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.269449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:19.284406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:19.285349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:19.286248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:19.286248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:19.286248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:19.286248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:19.286248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 412/817] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:19.286543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.313650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.353144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.363734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38aaf58-02d1-4ea1-9c7f-ba4f92846f0e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.387419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44186bda-f74c-4331-a07c-562f8e4de317 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.396465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.397086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.397360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.397631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.398309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] RBAC: Adding query filter params (type=network) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:19.398614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.400029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.404777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 406/818] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/services?type=network => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:19.405552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.423409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.424055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.424331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.424601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.425293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:19.425598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.426983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.438517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 413/819] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:19.439322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.506288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcfc24a-1b7b-40a9-880f-b373c39d38c6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.518909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.537830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a98fe-1142-40a7-957f-4afbe42580ac None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.539201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.539886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.540165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.540401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.547249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.548706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:19.560020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:19.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:19.561829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 407/820] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:19.562315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.576008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.576639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.576920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.577216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.577883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:19.578186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.578800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.579598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.584814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 414/821] 10.222.0.42 () {56 vars in 1124 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:19.585610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.670081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff575ff1-36fd-4c89-b640-6da3e27bba8d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.691874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1067377c-5edb-43a6-9c46-0cdbbea4e87e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.706332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.706992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.707274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.707547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.708251 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:19.708555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.709942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.721993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 408/822] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:19.722824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.728698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.729320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.729597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.729848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.730598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.732012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.754635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 415/823] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:19 2022] POST /identity/v3/endpoints => generated 327 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:19.755521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.758307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.821703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:19.853401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d231281-542a-43d2-a271-ceb6de6643d5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.892437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c7fb4-8dc4-4c60-9334-e5dc8311dd5d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:19.893559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.894264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.894506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.894777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.903943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.905443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:19.917879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:19.918849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:19.919788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 409/824] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:19.920700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:19.937914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:19.938554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:19.938838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:19.939147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:19.939921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:19.940233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:19.941975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:19.955759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 416/825] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:19.956414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:20.024101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:20.037043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7626f67-2832-4747-8e29-70049ac19b49 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:20.065229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78167582-d658-452a-84a3-734ec95cee86 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:20.070205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:20.070827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:20.071142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:20.071414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:20.072141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:20.073494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:20.078493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:20.095476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 410/826] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:19 2022] POST /identity/v3/endpoints => generated 346 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:20.096300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:20.104712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:20.105546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:20.105981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:20.106155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:20.113997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:20.115552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:20.128428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:20.129450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:20.130338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 417/827] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:20.131425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:20.223866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4814376c-5cd2-4bc6-8ece-e1cd4d2c6067 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:20.253781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b056f-432f-46c2-bb25-a9c08ed4d685 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:20.270018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:20.270192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:20.270863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:20.271168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:20.271489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:20.272277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:20.272579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:20.274100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:20.287381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 411/828] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:20.288594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:20.300442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:20.301159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:20.301437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:20.302068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/b211db19066a4767b09f1fcae194085a/roles/262cff71501a4fd3b5d8663bc982bde7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:20.320483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=b211db19066a4767b09f1fcae194085a, role_id=262cff71501a4fd3b5d8663bc982bde7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:20.322102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:20.342981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 418/829] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:58:20 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/b211db19066a4767b09f1fcae194085a/roles/262cff71501a4fd3b5d8663bc982bde7 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:58:20.342981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:20.359059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:20.433544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ef3d8c-8618-441a-ab91-b6df3aa5d412 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:20.467810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8725de1-5ded-4aae-9b5c-471f295fb06a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:20.481808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:20.482616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:20.483360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:20.483890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:20.484910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:20.485377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:20.487887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:20.507218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 412/830] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:20.510566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:20.517376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173b32e5-ed19-4475-9bcf-8514be52d0eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:20.518099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b32e5-ed19-4475-9bcf-8514be52d0eb None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:20.518451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b32e5-ed19-4475-9bcf-8514be52d0eb None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:20.518790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173b32e5-ed19-4475-9bcf-8514be52d0eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:20.519707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b32e5-ed19-4475-9bcf-8514be52d0eb None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=e2c09978fa3f44d895cbc17efdca2b32) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:20.520142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b32e5-ed19-4475-9bcf-8514be52d0eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:20.521626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173b32e5-ed19-4475-9bcf-8514be52d0eb None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:20.529929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 419/831] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:20 2022] GET /identity/v3/role_assignments?user.id=e2c09978fa3f44d895cbc17efdca2b32&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:20.632788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9acf717-2f2e-47dd-b3ef-cafc6ebe144d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:20.645368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:20.672509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0542ca3b-0cfb-422e-a387-b577eb4035b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:20.673208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0542ca3b-0cfb-422e-a387-b577eb4035b6 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:20.673486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0542ca3b-0cfb-422e-a387-b577eb4035b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:20.673760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0542ca3b-0cfb-422e-a387-b577eb4035b6 None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/afebddca912546119eea61f568bbd60d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:20.676161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0542ca3b-0cfb-422e-a387-b577eb4035b6 None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=afebddca912546119eea61f568bbd60d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:20.677653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0542ca3b-0cfb-422e-a387-b577eb4035b6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:20.694185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 413/832] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:58:20 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/afebddca912546119eea61f568bbd60d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:58:22.160819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734dfa8-3f82-4662-ac33-79d10dcd30fd None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.161419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734dfa8-3f82-4662-ac33-79d10dcd30fd None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.161419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734dfa8-3f82-4662-ac33-79d10dcd30fd None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.162928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 420/833] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:22.168562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99948fa-8890-4430-8915-8f5570410fa2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.168996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99948fa-8890-4430-8915-8f5570410fa2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.169095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99948fa-8890-4430-8915-8f5570410fa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.188020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a99948fa-8890-4430-8915-8f5570410fa2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:22.189259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a99948fa-8890-4430-8915-8f5570410fa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:22.256978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 414/834] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:22 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:22.278512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f734dfa8-3f82-4662-ac33-79d10dcd30fd None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:22.313790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:22.394118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f734dfa8-3f82-4662-ac33-79d10dcd30fd None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:22.419405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791537-dbb9-43b7-9665-537fcd5ffd0a None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.419722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791537-dbb9-43b7-9665-537fcd5ffd0a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.419953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791537-dbb9-43b7-9665-537fcd5ffd0a None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.421524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 415/835] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:22.426821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e3b24f-13b5-44cf-b6fc-f3583e8bd161 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.427135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e3b24f-13b5-44cf-b6fc-f3583e8bd161 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.427499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e3b24f-13b5-44cf-b6fc-f3583e8bd161 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.429174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:22.429833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.430116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.430394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.431280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:22.432769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:22.442560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:22.443623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:22.444467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 421/836] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:22 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:22.448044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c9e3b24f-13b5-44cf-b6fc-f3583e8bd161 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:22.449375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c9e3b24f-13b5-44cf-b6fc-f3583e8bd161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:22.451934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:22.537827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 416/837] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:22 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:22.557406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:22.563238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e3b24f-13b5-44cf-b6fc-f3583e8bd161 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:22.566778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab5eca9-1902-4390-87da-d69490deb736 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:22.596621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:22.604830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486248d4-6ffc-47cb-8078-f92501a62eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:22.605490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486248d4-6ffc-47cb-8078-f92501a62eb2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.605770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486248d4-6ffc-47cb-8078-f92501a62eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.606044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486248d4-6ffc-47cb-8078-f92501a62eb2 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.606739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486248d4-6ffc-47cb-8078-f92501a62eb2 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:22.607072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486248d4-6ffc-47cb-8078-f92501a62eb2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:22.608981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486248d4-6ffc-47cb-8078-f92501a62eb2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:22.619328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 422/838] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:22 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:22.619328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9611dd9a-642f-4d5a-9268-b2c7c6bcd2e8 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.619328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9611dd9a-642f-4d5a-9268-b2c7c6bcd2e8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.619328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9611dd9a-642f-4d5a-9268-b2c7c6bcd2e8 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.619328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 423/839] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:22.622830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9611dd9a-642f-4d5a-9268-b2c7c6bcd2e8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:22.682966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e3b24f-13b5-44cf-b6fc-f3583e8bd161 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:22.725770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0944b443-8bd7-41e1-86fa-eb50db660eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:22.726426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0944b443-8bd7-41e1-86fa-eb50db660eeb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.726701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0944b443-8bd7-41e1-86fa-eb50db660eeb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.727160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0944b443-8bd7-41e1-86fa-eb50db660eeb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.734306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9611dd9a-642f-4d5a-9268-b2c7c6bcd2e8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:22.738090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0944b443-8bd7-41e1-86fa-eb50db660eeb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-0944b443-8bd7-41e1-86fa-eb50db660eeb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:22.743052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:22.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:22.744832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 417/840] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 02:58:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:22.745537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47e4230-c010-4af0-8b87-b202aa7c0876 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.746714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47e4230-c010-4af0-8b87-b202aa7c0876 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.746714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47e4230-c010-4af0-8b87-b202aa7c0876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.763780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c47e4230-c010-4af0-8b87-b202aa7c0876 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:22.764967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c47e4230-c010-4af0-8b87-b202aa7c0876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:22.771599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add6c6fb-86bf-4a4b-ae6e-2f497e4b313c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:22.772232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add6c6fb-86bf-4a4b-ae6e-2f497e4b313c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.772513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add6c6fb-86bf-4a4b-ae6e-2f497e4b313c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.772787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add6c6fb-86bf-4a4b-ae6e-2f497e4b313c None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.773471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add6c6fb-86bf-4a4b-ae6e-2f497e4b313c None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:22.773777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add6c6fb-86bf-4a4b-ae6e-2f497e4b313c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:22.775187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add6c6fb-86bf-4a4b-ae6e-2f497e4b313c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:22.779948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 424/841] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:22 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:22.781143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca4883-12b0-4b74-9ae7-c924061b9185 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.781392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca4883-12b0-4b74-9ae7-c924061b9185 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.781666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ca4883-12b0-4b74-9ae7-c924061b9185 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.782875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 425/842] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:22.783669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca4883-12b0-4b74-9ae7-c924061b9185 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:22.809664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:22.856883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 418/843] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:22 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:22.856883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b33865-cc1d-49e6-a159-33d05e1daa6f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.856883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b33865-cc1d-49e6-a159-33d05e1daa6f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.856883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b33865-cc1d-49e6-a159-33d05e1daa6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.874199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-66b33865-cc1d-49e6-a159-33d05e1daa6f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:22.875410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-66b33865-cc1d-49e6-a159-33d05e1daa6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:22.892774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:22.915384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ca4883-12b0-4b74-9ae7-c924061b9185 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:22.955629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 419/844] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:22 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:22.957099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:22.957749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:22.958038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:22.958307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:22.959105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:22.959405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:22.960786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:22.963851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b33865-cc1d-49e6-a159-33d05e1daa6f None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:22.974773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 426/845] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 02:58:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:22.976238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.063161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:23.074690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b33865-cc1d-49e6-a159-33d05e1daa6f None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.083294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4244926c-29ee-4e49-8b8e-6d3aa5e14726 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.109886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.110530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.110812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.111136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.111777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.113173 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.121639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.122474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:23.122588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:23.123593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:23.129428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:23.130240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.130240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.130383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 420/846] 10.222.0.42 () {56 vars in 1132 bytes} [Tue Jul 26 02:58:22 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:23.130383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.135711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.137273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.145206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:23.153098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:23.154032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:23.154912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:23.154912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:23.154912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:23.154912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:23.154912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 427/847] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:23.155877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.245968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47db9c-713f-4c50-80ca-d07e0dfbc7d1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.263970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e168bbb-f1f2-4320-870d-b66520bbc952 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.283649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.284307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.284590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.284868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.295412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:23.299966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.300666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:23.300763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:23.301650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:23.302477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:23.303144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.303144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.303144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:23.303144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.304218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.305611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:23.307090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 421/848] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:23.307944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.315467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 428/849] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:23.316242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.385680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:23.412833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df853436-0d6e-443f-a64d-d288dfdee8fb None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.423693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd1121d-0241-4dea-8cb8-99630ef85a31 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.447686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.448318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.448600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.448965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.449674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:23.449985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.451405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.459505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.460428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.460878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.461294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.462146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:23.462594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.463318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 422/850] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:23.464148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.464581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.478179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 429/851] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:23.479269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.543769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:23.585697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615ad604-7f40-4e1c-bc8b-e7346182005a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.587807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807000de-e1c0-4f21-9d68-2ce6a0e53f97 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.624280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.624915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.625191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.625460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.626149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:23.626478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.626911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.627815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.627935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.628393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.628853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.629445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:23.634277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 423/852] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:23.635143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.640720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.642485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:23.660349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:23.661588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:23.662774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:23.671311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 430/853] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:23.673133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.758498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99acf086-5755-407b-b9aa-d508450b8a69 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.810055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.810425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.810919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.811252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.817948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cad64f9-7e09-4b54-a1b4-87321bf38170 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:23.831543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.833145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.849744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:23.857816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:23.858732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:23.859668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:23.859668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:23.859668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:23.859668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:23.864850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 424/854] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:23.870244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.875427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:23.876076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:23.876376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:23.876646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:23.877466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:23.877771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:23.879399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:23.905449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 431/855] 10.222.0.42 () {56 vars in 1149 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:23.910775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:23.947730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:24.023512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aed2ae-ed97-401e-b192-55c71f9b6bdd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.023785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fa0ae0-68fb-416e-9aa2-ec862a8a54a7 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.059035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.059778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.060054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.060319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.061051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:24.061354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.061887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.062746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.062880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.063458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.063924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.078992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 425/856] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:24.085076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:24.127214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:24.158965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.160683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.199882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb0efa0-3168-410c-b55c-ab71de77623d None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.213339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:24.247023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.247706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.247976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.248249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.261058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.262581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:24.279553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:24.280598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:24.284061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 426/857] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:24.286667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:24.312868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 432/858] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:23 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 26 02:58:24.315668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:24.402358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:24.426417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c47988-adae-4dbb-9da3-1b6eca0ec6c4 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.431800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3447d273-d565-4942-bad7-1d76dafe7ab8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.463171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.463884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.464234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.464607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.465578 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:24.465950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.467597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.484217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.485309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.485859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.486298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.492777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 427/859] 10.222.0.42 () {56 vars in 1135 bytes} [Tue Jul 26 02:58:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:24.493795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:24.501020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.502701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.523534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:24.526190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:24.527197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:24.528098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:24.528098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:24.528098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:24.528098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:24.534821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 433/860] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:24.536167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:24.612443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb439f5f-5c43-4722-8cfc-0e93f3baa35f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.652612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49fd561-6e18-4a07-af6e-8e6fa4c710dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.653892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49fd561-6e18-4a07-af6e-8e6fa4c710dd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.654345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49fd561-6e18-4a07-af6e-8e6fa4c710dd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.654784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49fd561-6e18-4a07-af6e-8e6fa4c710dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.661716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2864463-1c42-4541-b0ca-8139e179c3e4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.693091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:24.703730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.704638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.705088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.705987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.706896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:24.707411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.709607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.730459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 434/861] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:24.732061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:24.741733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49fd561-6e18-4a07-af6e-8e6fa4c710dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.743329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49fd561-6e18-4a07-af6e-8e6fa4c710dd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.773692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:24.850055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37748f5b-96eb-4665-bbe9-ed5ce7b18458 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.859218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 428/862] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:24 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:24.860313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a1d294-56c8-4304-8380-ea81cb4baf77 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.860548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a1d294-56c8-4304-8380-ea81cb4baf77 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.860858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a1d294-56c8-4304-8380-ea81cb4baf77 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.863000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 429/863] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:24.863000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a1d294-56c8-4304-8380-ea81cb4baf77 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:24.888228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3f2071-0a5f-4919-b27d-5464446d79db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:24.889005 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f2071-0a5f-4919-b27d-5464446d79db None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.889405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f2071-0a5f-4919-b27d-5464446d79db None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.889726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f2071-0a5f-4919-b27d-5464446d79db None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.896752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3f2071-0a5f-4919-b27d-5464446d79db None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:24.898211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3f2071-0a5f-4919-b27d-5464446d79db None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7c3f2071-0a5f-4919-b27d-5464446d79db None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:24.910124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:24.911208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:24.912281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:24.916096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 435/864] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:24.917221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed27a46-6815-4286-9d57-7e9a242f971b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:24.917542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed27a46-6815-4286-9d57-7e9a242f971b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:24.917821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed27a46-6815-4286-9d57-7e9a242f971b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:24.935592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-9ed27a46-6815-4286-9d57-7e9a242f971b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:24.936801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9ed27a46-6815-4286-9d57-7e9a242f971b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:24.973593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a1d294-56c8-4304-8380-ea81cb4baf77 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:24.985647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:25.001481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:25.013450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 436/865] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:24 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:25.014306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed27a46-6815-4286-9d57-7e9a242f971b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.015885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.016540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.016815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.017089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.030629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.032407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:25.047845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:25.048817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:25.049724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:25.049724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:25.049724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:25.049724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:25.049724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 430/866] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:25.050015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.130358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed27a46-6815-4286-9d57-7e9a242f971b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.165394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfc7f51-d505-4438-8c48-fb88040fec22 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.167836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.168463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.168740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.169015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.169775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:25.170078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.171542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.183939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 437/867] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:25.184825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.214995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.215569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.215845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.216111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.216804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.218229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.230184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:25.246978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 431/868] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:25 2022] POST /identity/v3/services => generated 251 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:25.246978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.301348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:25.329618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e699459-fc57-4386-9939-6c5726dc3a7f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.381959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb38690b-b771-4432-97b4-fef8ae9c80c2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.384804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.386560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.386840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.387142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.387856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:25.388189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.389805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.411125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 438/869] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:25.415538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.432164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.432817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.433133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.433409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.434215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=b211db19066a4767b09f1fcae194085a) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:25.434564 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.436069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.442331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 432/870] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:25 2022] GET /identity/v3/role_assignments?user.id=b211db19066a4767b09f1fcae194085a&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 608 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:25.543514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efac5d85-7364-41c1-8f04-900463c7ca3f None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.546429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:25.586900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b832d0-e671-418f-82dd-eb83b548cc3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.587615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b832d0-e671-418f-82dd-eb83b548cc3c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.587899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b832d0-e671-418f-82dd-eb83b548cc3c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.588176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b832d0-e671-418f-82dd-eb83b548cc3c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.596927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b832d0-e671-418f-82dd-eb83b548cc3c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.598425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b832d0-e671-418f-82dd-eb83b548cc3c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-28b832d0-e671-418f-82dd-eb83b548cc3c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:25.611423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:25.612849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:25.614181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 439/871] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:25.618544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.671062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8de3972-cc40-4288-af49-2c0b7c9aa8c4 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.671354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8de3972-cc40-4288-af49-2c0b7c9aa8c4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.671577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8de3972-cc40-4288-af49-2c0b7c9aa8c4 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.673440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 440/872] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:58:25.680310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abf0465-37b1-40ea-aa35-ea2367dfd344 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.680591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abf0465-37b1-40ea-aa35-ea2367dfd344 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.680874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abf0465-37b1-40ea-aa35-ea2367dfd344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.702132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1abf0465-37b1-40ea-aa35-ea2367dfd344 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:25.703432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1abf0465-37b1-40ea-aa35-ea2367dfd344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:25.734748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:25.737776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0954b7d5-8581-4c29-bae3-ea61a6ef9da8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.777731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.778377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.778666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.778935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.779754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:25.780066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.781566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.791061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 441/873] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:25 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:25.797698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 433/874] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:25.803735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abf0465-37b1-40ea-aa35-ea2367dfd344 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.812891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:25.911072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:25.926619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ce81b-5906-463f-bc99-0ddecb974d98 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.935289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abf0465-37b1-40ea-aa35-ea2367dfd344 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:25.972051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e576bad8-aa4d-4920-b3e3-cba35ff74be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.972389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc2c2b3-9e4e-47f6-b4d1-7e03382bce8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:25.972937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2c2b3-9e4e-47f6-b4d1-7e03382bce8b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.973211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2c2b3-9e4e-47f6-b4d1-7e03382bce8b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.973482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2c2b3-9e4e-47f6-b4d1-7e03382bce8b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.973864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e576bad8-aa4d-4920-b3e3-cba35ff74be9 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:25.974277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2c2b3-9e4e-47f6-b4d1-7e03382bce8b None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=afebddca912546119eea61f568bbd60d) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:25.974550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e576bad8-aa4d-4920-b3e3-cba35ff74be9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:25.974649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2c2b3-9e4e-47f6-b4d1-7e03382bce8b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.975131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e576bad8-aa4d-4920-b3e3-cba35ff74be9 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:25.976049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e576bad8-aa4d-4920-b3e3-cba35ff74be9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:25.976150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2c2b3-9e4e-47f6-b4d1-7e03382bce8b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.977712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e576bad8-aa4d-4920-b3e3-cba35ff74be9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:25.990276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 442/875] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:25 2022] GET /identity/v3/role_assignments?user.id=afebddca912546119eea61f568bbd60d&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:26.005040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 434/876] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:25 2022] POST /identity/v3/services => generated 269 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:58:27.407777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb6ccd4-07b7-4eca-a30c-8e41075fbb81 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:27.408813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb6ccd4-07b7-4eca-a30c-8e41075fbb81 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:27.409293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb6ccd4-07b7-4eca-a30c-8e41075fbb81 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:27.411168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 443/877] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:27.416643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8f5c54-9f4d-4b7b-a302-1f976bcd90c9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:27.417507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8f5c54-9f4d-4b7b-a302-1f976bcd90c9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:27.417996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8f5c54-9f4d-4b7b-a302-1f976bcd90c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:27.436759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-aa8f5c54-9f4d-4b7b-a302-1f976bcd90c9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:27.438254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-aa8f5c54-9f4d-4b7b-a302-1f976bcd90c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:27.454218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:27.519400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 435/878] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:27 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:27.542357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb6ccd4-07b7-4eca-a30c-8e41075fbb81 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:27.595336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733e232-0a43-4299-97d3-38784621ded6 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:27.595671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733e232-0a43-4299-97d3-38784621ded6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:27.595876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733e232-0a43-4299-97d3-38784621ded6 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:27.597517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 436/879] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:58:27.602522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf34ace0-28af-42cf-939f-35574d6f395c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:27.602812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf34ace0-28af-42cf-939f-35574d6f395c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:27.603223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf34ace0-28af-42cf-939f-35574d6f395c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:27.622768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-bf34ace0-28af-42cf-939f-35574d6f395c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:27.624027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bf34ace0-28af-42cf-939f-35574d6f395c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:27.664646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb6ccd4-07b7-4eca-a30c-8e41075fbb81 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:27.712792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 437/880] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:27 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:27.716773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:27.717686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:27.718159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:27.718627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:27.719820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:27.721391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:27.731084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:27.732100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:27.732949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:27.733701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf34ace0-28af-42cf-939f-35574d6f395c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:27.736218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 444/881] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:27 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:27.743983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:27.761693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:27.824611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:27.850881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a293f2be-26da-471a-8830-f53800dba25a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:27.852062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf34ace0-28af-42cf-939f-35574d6f395c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:27.887209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:27.888101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:27.888567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:27.889029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:27.889909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:27.890312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:27.890477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:27.891308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:27.891727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:27.892098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:27.892361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:27.899070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 445/882] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:27 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:27.903598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:27.910095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:27.915609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:27.916616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:27.917525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:27.924924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 438/883] 10.222.0.42 () {56 vars in 1120 bytes} [Tue Jul 26 02:58:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:27.926286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:27.989921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.025551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f301348-7ae5-4a57-9595-ad9e515e5d21 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.046977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa480db3-1a2e-4afc-956d-e96895dbb2d5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.067583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8762ac-5ccd-4918-9127-e6bc4d09c2bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.068224 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8762ac-5ccd-4918-9127-e6bc4d09c2bf None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.068510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8762ac-5ccd-4918-9127-e6bc4d09c2bf None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.068787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8762ac-5ccd-4918-9127-e6bc4d09c2bf None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.069485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8762ac-5ccd-4918-9127-e6bc4d09c2bf None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:28.069795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8762ac-5ccd-4918-9127-e6bc4d09c2bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.071218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8762ac-5ccd-4918-9127-e6bc4d09c2bf None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.076602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 446/884] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:27 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:28.077807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b5077-1f3a-4ba7-b25f-c06725d06fb7 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.078124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b5077-1f3a-4ba7-b25f-c06725d06fb7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.078379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b5077-1f3a-4ba7-b25f-c06725d06fb7 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.083608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 447/885] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:28.083608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3752ac-b9fc-48cb-9d42-a9dafad4694d None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.083608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3752ac-b9fc-48cb-9d42-a9dafad4694d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.083608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3752ac-b9fc-48cb-9d42-a9dafad4694d None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.083608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 448/886] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:28.083608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3752ac-b9fc-48cb-9d42-a9dafad4694d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.088708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b807f98-3fcb-4ed9-990b-9af22857ba60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.089436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b807f98-3fcb-4ed9-990b-9af22857ba60 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.089714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b807f98-3fcb-4ed9-990b-9af22857ba60 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.089985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b807f98-3fcb-4ed9-990b-9af22857ba60 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.090718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b807f98-3fcb-4ed9-990b-9af22857ba60 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:28.091066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b807f98-3fcb-4ed9-990b-9af22857ba60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.092474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b807f98-3fcb-4ed9-990b-9af22857ba60 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.095464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.104611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 439/887] 10.222.0.42 () {56 vars in 1115 bytes} [Tue Jul 26 02:58:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:28.105987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ad5188-0abb-4b61-acf6-513d2ac93835 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.106275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ad5188-0abb-4b61-acf6-513d2ac93835 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.106552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ad5188-0abb-4b61-acf6-513d2ac93835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.125096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-58ad5188-0abb-4b61-acf6-513d2ac93835 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:28.126322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-58ad5188-0abb-4b61-acf6-513d2ac93835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:28.192211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3752ac-b9fc-48cb-9d42-a9dafad4694d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.195067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 440/888] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:28 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:28.196437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37204dc-bc14-4a84-8e0c-e6bcb582cf70 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.196862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37204dc-bc14-4a84-8e0c-e6bcb582cf70 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.196958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37204dc-bc14-4a84-8e0c-e6bcb582cf70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.215304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a37204dc-bc14-4a84-8e0c-e6bcb582cf70 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:28.217180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a37204dc-bc14-4a84-8e0c-e6bcb582cf70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:28.240538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.241178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.241476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.241752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.242527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] RBAC: Adding query filter params (interface=public, service_id=4a251dca73574b8e801afac4fdaa89b5, region_id=RegionOne) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:28.242836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.244278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.249691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 449/889] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/endpoints?service_id=4a251dca73574b8e801afac4fdaa89b5&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:28.250527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.261703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.293971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 441/890] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:28 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:28.294937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37204dc-bc14-4a84-8e0c-e6bcb582cf70 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.354121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.361967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a058767-1fd1-4658-a3ec-efc4d11486dd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.396148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37204dc-bc14-4a84-8e0c-e6bcb582cf70 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.396889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.397549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.397829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.398096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.408776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.410243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:28.424698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:28.425625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:28.426504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:28.426504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:28.426504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:28.426504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:28.426504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 450/891] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:28.426786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.436341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.436992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.437275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.437900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.438542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.439964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:28.448976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:28.449940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:28.450809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:28.455275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 442/892] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:28.456371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.484592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.526080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ace10-0aa8-43c9-856f-314e8b19c3b1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.560726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3dc91f2-5ab8-4f30-a191-cd0913c393f6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.562379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06932568-6130-498e-aef1-e1694842e574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.563032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06932568-6130-498e-aef1-e1694842e574 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.563308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06932568-6130-498e-aef1-e1694842e574 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.563584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06932568-6130-498e-aef1-e1694842e574 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.564248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06932568-6130-498e-aef1-e1694842e574 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.565628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06932568-6130-498e-aef1-e1694842e574 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.583019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 451/893] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/services => generated 1268 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 02:58:28.583019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06932568-6130-498e-aef1-e1694842e574 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.586027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.601919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.602621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.602906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.603205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.603858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.605699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:28.616125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:28.617325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:28.618191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 443/894] 10.222.0.42 () {56 vars in 1132 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:28.619699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.694389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06932568-6130-498e-aef1-e1694842e574 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.721845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.738282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.738942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.739267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.739546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.739921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b5f588-4d37-4448-b160-f8cad6bce1b1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.740258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:28.740567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.741956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.756049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 452/895] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:28.756676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.776940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.777948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.778390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.778809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.779680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:28.780130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.781683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.786904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 444/896] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:28.787955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.842382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:28.872594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddfa210-4d14-4a37-b658-3e992edda596 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.898111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c337a064-43bb-4008-8fc3-04339a1ee9e9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:28.909907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.910544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.910826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.911134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.911835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:28.912143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.913615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.918434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 453/897] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:28.919271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.942147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:28.942800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:28.943119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:28.943390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:28.952429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:28.953997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:28.965434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:28.966400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 445/898] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:28.967473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:28.975039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:29.044831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75572955-fd31-4d94-8519-77604b843d7b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.081647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be556d4-7ece-4790-a1d8-46391456676b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.084788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.084788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.085036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.085230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.085931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:29.086240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.087662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:29.095010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 454/899] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:29.095010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:29.097211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:29.130293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.131390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.132255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.132813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.133808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:29.134268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.136746 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:29.149013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 446/900] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:28 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:29.153710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:29.222417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d1b6b2-d859-48b4-8501-e3da4b24ec7e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.235979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:29.268132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.268771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.269165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.269446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.270264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:29.270577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.273755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:29.289610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608cbbc0-4b22-4ef5-b5a1-d66ebca82ad5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.298786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 455/901] 10.222.0.42 () {56 vars in 1149 bytes} [Tue Jul 26 02:58:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:29.304451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:29.325419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.326366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.326854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.327336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.328285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] RBAC: Adding query filter params (interface=public, service_id=05f940c6d5fd4b13b6bb0b7cb3d96c74, region_id=RegionOne) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:29.328775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.330359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:29.340582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 447/902] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 02:58:29 2022] GET /identity/v3/endpoints?service_id=05f940c6d5fd4b13b6bb0b7cb3d96c74&interface=public®ion_id=RegionOne => generated 194 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:29.342130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:29.383134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:29.410083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7387f60c-4fbe-41cc-a358-7e957e3589fd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.445218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.445843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.446123 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.446396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.448798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b8507-e393-45ed-bb5d-8dec8b3e976a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.483414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:29.485783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.486686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.487186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.487632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.488513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.490137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:29.496527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 448/903] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:29 2022] GET /identity/v3/services => generated 1268 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 02:58:29.522825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.524435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:29.653080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 456/904] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:29 2022] GET /identity/v3/auth/tokens => generated 1362 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 26 02:58:29.660224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:29.780527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f559c5-2030-4d1c-87e8-426cf13eed03 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.782040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:29.816757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.817405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.817680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.817951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.830125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.831714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:29.847100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:29.848029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:29.849327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:29.849327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:29.849327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:29.849327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:29.849327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 449/905] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:29.854283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:29.931367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:29.956521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d24c4-a2f3-4933-9fb7-5c9e0a0fc5e4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:29.993794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:29.994496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:29.994776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:29.995086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:29.995987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:29.996277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:29.997700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:30.012026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 457/906] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:30.017744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:30.127446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ab8f57-6359-40be-9333-a3196cb0bf8e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:30.163151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:30.164076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:30.164492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:30.164901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:30.172024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:30.173635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:30.185795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:30.186888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:30.188004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:30.190063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 450/907] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:30.196140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:30.306127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd313fbd-e7a6-4c6b-a1f9-a96d70123696 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:30.337993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:30.348593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b861a60a-7424-497b-ad78-b382048c6f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:30.349538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861a60a-7424-497b-ad78-b382048c6f06 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:30.349997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861a60a-7424-497b-ad78-b382048c6f06 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:30.350465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b861a60a-7424-497b-ad78-b382048c6f06 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:30.351477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b861a60a-7424-497b-ad78-b382048c6f06 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:30.352014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b861a60a-7424-497b-ad78-b382048c6f06 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:30.353641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b861a60a-7424-497b-ad78-b382048c6f06 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:30.367688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 458/908] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:30.374125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:30.391731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:30.487277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b1ba1d-77d3-4cc9-a623-e9aef908070a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:30.521346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb85e2d-1757-4605-ab5a-fb90d89fb126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:30.521976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb85e2d-1757-4605-ab5a-fb90d89fb126 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:30.522258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb85e2d-1757-4605-ab5a-fb90d89fb126 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:30.522536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb85e2d-1757-4605-ab5a-fb90d89fb126 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:30.523367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb85e2d-1757-4605-ab5a-fb90d89fb126 None admin] RBAC: Adding query filter params (role.id=f82ad6f8506b4ef089846400682f59f2, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=b211db19066a4767b09f1fcae194085a) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:30.523681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb85e2d-1757-4605-ab5a-fb90d89fb126 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:30.525129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb85e2d-1757-4605-ab5a-fb90d89fb126 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:30.531098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 451/909] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:30 2022] GET /identity/v3/role_assignments?user.id=b211db19066a4767b09f1fcae194085a&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=f82ad6f8506b4ef089846400682f59f2 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:30.630771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df9e3fe-814a-49f1-82d3-bf1ef1db1233 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:30.631440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df9e3fe-814a-49f1-82d3-bf1ef1db1233 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:30.631440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df9e3fe-814a-49f1-82d3-bf1ef1db1233 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:30.633017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 459/910] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:30.638742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb7f49-161a-4d0f-b9b9-db7510131d00 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:30.639060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb7f49-161a-4d0f-b9b9-db7510131d00 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:30.639338 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cb7f49-161a-4d0f-b9b9-db7510131d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:30.659478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c1cb7f49-161a-4d0f-b9b9-db7510131d00 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:30.660634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c1cb7f49-161a-4d0f-b9b9-db7510131d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:30.734542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 452/911] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:30 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:30.757420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df9e3fe-814a-49f1-82d3-bf1ef1db1233 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:30.871194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df9e3fe-814a-49f1-82d3-bf1ef1db1233 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:30.907499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:30.908154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:30.908437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:30.908711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:30.909351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:30.910787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:30.920150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:30.921145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:30.921968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 460/912] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:30 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:30.927445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb7f49-161a-4d0f-b9b9-db7510131d00 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:30.954478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:31.047442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cb7f49-161a-4d0f-b9b9-db7510131d00 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:31.084204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9a9bde-3b37-49dd-88d0-6ebc82d234bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:31.084838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a9bde-3b37-49dd-88d0-6ebc82d234bc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.085119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a9bde-3b37-49dd-88d0-6ebc82d234bc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.085394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9a9bde-3b37-49dd-88d0-6ebc82d234bc None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.086086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a9bde-3b37-49dd-88d0-6ebc82d234bc None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:31.086392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a9bde-3b37-49dd-88d0-6ebc82d234bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:31.087816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9a9bde-3b37-49dd-88d0-6ebc82d234bc None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:31.093073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 453/913] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:30 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:31.099110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:31.156206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:31.208600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3804326-e19b-40a1-a4c3-6d46cf8adc92 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.208797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72563395-1ea0-4776-84d1-042cbfd4894c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:31.208880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3804326-e19b-40a1-a4c3-6d46cf8adc92 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.209088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3804326-e19b-40a1-a4c3-6d46cf8adc92 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.211746 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 454/914] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:58:31.216013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e61ef8-3640-46b3-82ad-0f6a11a4c496 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.216302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e61ef8-3640-46b3-82ad-0f6a11a4c496 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.216579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e61ef8-3640-46b3-82ad-0f6a11a4c496 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.236476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-11e61ef8-3640-46b3-82ad-0f6a11a4c496 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:31.238062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-11e61ef8-3640-46b3-82ad-0f6a11a4c496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:31.244882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:31.245554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.245836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.246112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.246807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:31.247157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:31.248538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:31.254013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 461/915] 10.222.0.42 () {56 vars in 1114 bytes} [Tue Jul 26 02:58:31 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:31.260686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:31.326020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 455/916] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:31 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:31.348894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e61ef8-3640-46b3-82ad-0f6a11a4c496 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:31.351136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:31.379500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d38d8-df00-4d76-a608-296adb0c5b5d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:31.397617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:31.418670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d483b187-4584-4735-9fe3-e0ba24a4550d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:31.419743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d483b187-4584-4735-9fe3-e0ba24a4550d None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.420387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d483b187-4584-4735-9fe3-e0ba24a4550d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.420902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d483b187-4584-4735-9fe3-e0ba24a4550d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.421949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d483b187-4584-4735-9fe3-e0ba24a4550d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:31.423698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d483b187-4584-4735-9fe3-e0ba24a4550d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:31.455287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 462/917] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:31 2022] POST /identity/v3/endpoints => generated 346 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:58:31.473130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e61ef8-3640-46b3-82ad-0f6a11a4c496 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:31.518495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a922ae91-165e-4aa1-aab3-a41431f7cb83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:31.519182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a922ae91-165e-4aa1-aab3-a41431f7cb83 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.519570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a922ae91-165e-4aa1-aab3-a41431f7cb83 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.519846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a922ae91-165e-4aa1-aab3-a41431f7cb83 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.520699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a922ae91-165e-4aa1-aab3-a41431f7cb83 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:31.522116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a922ae91-165e-4aa1-aab3-a41431f7cb83 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:31.542063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 456/918] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:31 2022] POST /identity/v3/services => generated 241 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:58:31.726525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebffa9c-c847-4ab0-b476-08f5bf54cafb None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.727131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebffa9c-c847-4ab0-b476-08f5bf54cafb None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.727131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebffa9c-c847-4ab0-b476-08f5bf54cafb None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.728673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 463/919] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:31.733994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a43eff-1c42-42c1-bba4-20d8a4e2ef08 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:31.734284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a43eff-1c42-42c1-bba4-20d8a4e2ef08 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:31.734561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a43eff-1c42-42c1-bba4-20d8a4e2ef08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:31.754924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c4a43eff-1c42-42c1-bba4-20d8a4e2ef08 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:31.756176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c4a43eff-1c42-42c1-bba4-20d8a4e2ef08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:31.817562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:31.838886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 457/920] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:31 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:31.862039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebffa9c-c847-4ab0-b476-08f5bf54cafb None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:31.972906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebffa9c-c847-4ab0-b476-08f5bf54cafb None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:32.009957 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:32.011034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:32.011559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:32.012056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:32.012951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:32.015247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:32.025046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:32.026094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 26 02:58:32.027068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:32.030058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 464/921] 10.222.0.42 () {56 vars in 1132 bytes} [Tue Jul 26 02:58:31 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:32.039004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a43eff-1c42-42c1-bba4-20d8a4e2ef08 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:32.141427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a43eff-1c42-42c1-bba4-20d8a4e2ef08 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:32.176827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:32.177470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:32.177754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:32.178028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:32.178731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:32.179062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:32.180459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:32.185995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 458/922] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:32 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:32.191799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:32.229695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:32.297730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c64fd15-09cf-4272-8ae2-ef3185bbf986 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:32.333773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b0fbd2-518d-407a-b257-580fc30c1eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:32.334456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0fbd2-518d-407a-b257-580fc30c1eb0 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:32.334742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0fbd2-518d-407a-b257-580fc30c1eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:32.335039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0fbd2-518d-407a-b257-580fc30c1eb0 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:32.335734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b0fbd2-518d-407a-b257-580fc30c1eb0 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:32.336046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b0fbd2-518d-407a-b257-580fc30c1eb0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:32.337473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b0fbd2-518d-407a-b257-580fc30c1eb0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:32.347135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 465/923] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:32 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:32.349185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:32.401262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:32.457123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836b88d3-b471-4f98-8f99-9f18c5c08fc5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:32.494722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7c459c-bcab-4d2b-b89c-2b0df8071744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:32.495421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c459c-bcab-4d2b-b89c-2b0df8071744 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:32.495739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c459c-bcab-4d2b-b89c-2b0df8071744 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:32.496019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c459c-bcab-4d2b-b89c-2b0df8071744 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:32.496758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7c459c-bcab-4d2b-b89c-2b0df8071744 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:32.498147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7c459c-bcab-4d2b-b89c-2b0df8071744 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:32.521457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 459/924] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:32 2022] POST /identity/v3/endpoints => generated 347 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:32.624957 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f374c521-8736-4f90-ab3a-099cf340e02c None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:32.625319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f374c521-8736-4f90-ab3a-099cf340e02c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:32.625488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f374c521-8736-4f90-ab3a-099cf340e02c None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:32.627237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 466/925] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:32.632934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e417b5-1a52-4e66-8cbc-48a31fbdff7c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:32.633230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e417b5-1a52-4e66-8cbc-48a31fbdff7c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:32.633500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e417b5-1a52-4e66-8cbc-48a31fbdff7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:32.654289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-60e417b5-1a52-4e66-8cbc-48a31fbdff7c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:32.655579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-60e417b5-1a52-4e66-8cbc-48a31fbdff7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:32.743547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 460/926] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:32 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:32.772036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f374c521-8736-4f90-ab3a-099cf340e02c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:32.895849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:32.897532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f374c521-8736-4f90-ab3a-099cf340e02c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:32.933491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:32.934150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:32.934435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:32.934710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:32.945389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:32.950667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:32.951639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:32.952499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 467/927] 10.222.0.42 () {56 vars in 1120 bytes} [Tue Jul 26 02:58:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:32.959940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e417b5-1a52-4e66-8cbc-48a31fbdff7c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:32.989861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:33.065428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e417b5-1a52-4e66-8cbc-48a31fbdff7c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:33.101132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:33.102128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.102594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.103139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.104117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:33.104638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:33.106811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:33.116183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 461/928] 10.222.0.42 () {56 vars in 1115 bytes} [Tue Jul 26 02:58:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:33.122887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:33.233197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcdac5e-7954-4d9e-9678-310a47731114 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:33.269074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:33.270098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.270577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.271156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.282512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:33.284359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:33.300708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:33.301662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:33.302545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:33.302545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:33.302545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:33.302545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:33.304832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 468/929] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:33.310993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:33.409853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:33.419176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db67bfb-48a3-4349-93a2-067854425069 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:33.453446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:33.454113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.454400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.454675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.455406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:33.455712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:33.457101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:33.473718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 462/930] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:33.479394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:33.495785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:33.588173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2e85b-ccf4-4c07-b12d-ff136f7da316 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:33.623047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2a9a8c-ba97-4495-9592-372964883a51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:33.623711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2a9a8c-ba97-4495-9592-372964883a51 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.623997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2a9a8c-ba97-4495-9592-372964883a51 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.624274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2a9a8c-ba97-4495-9592-372964883a51 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.632811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2a9a8c-ba97-4495-9592-372964883a51 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:33.634315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2a9a8c-ba97-4495-9592-372964883a51 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-6f2a9a8c-ba97-4495-9592-372964883a51 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:33.646693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:33.647731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:33.648672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 469/931] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 02:58:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:33.654718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:33.676847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76eb5e8-aa4d-4bf7-8366-ef45326b0c4f None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.677060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76eb5e8-aa4d-4bf7-8366-ef45326b0c4f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.677279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76eb5e8-aa4d-4bf7-8366-ef45326b0c4f None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.678859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 470/932] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:33.684373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecddaa33-c07d-4a86-bb80-0c6e1800831f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.684715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecddaa33-c07d-4a86-bb80-0c6e1800831f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.684999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecddaa33-c07d-4a86-bb80-0c6e1800831f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.704554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ecddaa33-c07d-4a86-bb80-0c6e1800831f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:33.705773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ecddaa33-c07d-4a86-bb80-0c6e1800831f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:33.770735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c54ff0a-1eb2-4d4b-8abe-ea180de39ad2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:33.783272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 471/933] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:33 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:33.783272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7660fff-32d3-4e6b-a97b-ecbdc37b0a3e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.783272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7660fff-32d3-4e6b-a97b-ecbdc37b0a3e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.783945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7660fff-32d3-4e6b-a97b-ecbdc37b0a3e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.784877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 472/934] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:33.790017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2dfe3-c87f-48b6-85fc-e870d6a2e51e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.790346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2dfe3-c87f-48b6-85fc-e870d6a2e51e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.790537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2dfe3-c87f-48b6-85fc-e870d6a2e51e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.800075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:33.814919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:33.815365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-74c2dfe3-c87f-48b6-85fc-e870d6a2e51e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:33.815573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.815850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.816118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.816590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-74c2dfe3-c87f-48b6-85fc-e870d6a2e51e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:33.816918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:33.817224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:33.818665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:33.836823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 463/935] 10.222.0.42 () {56 vars in 1149 bytes} [Tue Jul 26 02:58:33 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:33.837618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:33.850905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:33.893761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 473/936] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:33 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:33.896405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c2dfe3-c87f-48b6-85fc-e870d6a2e51e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:33.943714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740bbbad-eb1b-4ea6-822e-f82b8c2972a4 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:33.978007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:33.978632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:33.978911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:33.979207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:33.979938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:33.981379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:33.984013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING keystone.server.flask.common [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 26 02:58:34.004659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.sql.core [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 26 02:58:34.004659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 26 02:58:34.004659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [parameters: {'id': '85c28cb078c64620a477c0c3bed949f1', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 26 02:58:34.004659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80428) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 85c28cb078c64620a477c0c3bed949f1.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 85c28cb078c64620a477c0c3bed949f1. Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result = self._query(query) Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application conn.query(q) Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 26 02:58:34.012897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result.read() Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 26 02:58:34.013738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application next(self.gen) Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application yield resource Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application next(self.gen) Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.session.rollback() Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.force_reraise() Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise self.value Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application session.commit() Jul 26 02:58:34.015234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._prepare_impl() Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.session.flush() Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._flush(objects) Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application compat.raise_( Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application flush_context.execute() Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 26 02:58:34.016146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rec.execute(self) Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 26 02:58:34.017048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result = self._query(query) Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application conn.query(q) Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result.read() Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 26 02:58:34.017842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [parameters: {'id': '85c28cb078c64620a477c0c3bed949f1', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.018685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:34.021620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 85c28cb078c64620a477c0c3bed949f1. Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 464/937] 10.222.0.42 () {60 vars in 1140 bytes} [Tue Jul 26 02:58:33 2022] POST /identity/v3/projects => generated 262 bytes in 182 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:34.025490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.027022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c2dfe3-c87f-48b6-85fc-e870d6a2e51e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.060277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.066541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.067223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.067504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.067778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.078491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.080041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:34.096477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:34.097403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:34.098296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:34.098296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:34.098296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:34.098296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:34.099741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.102862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 474/938] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:34.103763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.141666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ec68b-2ec5-432a-9c0b-629c83570581 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.177930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.178586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.178871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.179278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.179819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.181423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:34.190669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:34.191727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:34.192563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 465/939] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:34.193379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.217773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1f03d-1fbc-4165-afaf-0731e33b9033 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.257335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.257978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.258268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.258518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.265593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.267080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:34.278816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:34.279928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 475/940] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:34.280829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.307233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1039f511-bf9b-4383-8ae9-1496578f9280 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.329482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.338318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.345943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.346668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.347069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.347406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.348177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:34.348549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.349999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.369919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 466/941] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:34.369919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.394780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189d19d-2251-4614-b2bb-5781ee8ec196 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.431061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.431945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.432224 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.432497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.433352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:34.433945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.435685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.446969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 476/942] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:34.446969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.471676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc2577a-0514-49e0-bc92-f2be4edc4845 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.508161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.508798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.509095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.509366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.510108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:34.510414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.511895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.529503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 467/943] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:34.530332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.586695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a5473-7c50-47db-ad7e-72f2dc043779 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.599354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.603902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.630655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76385024-2316-488d-a268-b4126eaa4943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.631464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76385024-2316-488d-a268-b4126eaa4943 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.632204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76385024-2316-488d-a268-b4126eaa4943 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.632548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76385024-2316-488d-a268-b4126eaa4943 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.640861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76385024-2316-488d-a268-b4126eaa4943 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.642365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76385024-2316-488d-a268-b4126eaa4943 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-76385024-2316-488d-a268-b4126eaa4943 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:34.655373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:34.656354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:34.657268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 477/944] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:34.658441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af384452-c98c-420b-9592-6655ab363579 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.658441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af384452-c98c-420b-9592-6655ab363579 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.658604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af384452-c98c-420b-9592-6655ab363579 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.659999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 478/945] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:34.664302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af384452-c98c-420b-9592-6655ab363579 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.665860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8799598f-8cde-4928-ab42-65a34b0c74d2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.707575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32ed4c6-bae7-4702-95a1-c0084b64ae25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.708302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32ed4c6-bae7-4702-95a1-c0084b64ae25 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.708581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32ed4c6-bae7-4702-95a1-c0084b64ae25 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.708855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32ed4c6-bae7-4702-95a1-c0084b64ae25 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.709572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32ed4c6-bae7-4702-95a1-c0084b64ae25 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:34.709873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32ed4c6-bae7-4702-95a1-c0084b64ae25 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.711408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32ed4c6-bae7-4702-95a1-c0084b64ae25 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.717176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 468/946] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:34.720317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0b62e5-68a7-4e01-a712-c14e1f632415 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.720317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0b62e5-68a7-4e01-a712-c14e1f632415 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.720317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0b62e5-68a7-4e01-a712-c14e1f632415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.740646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6b0b62e5-68a7-4e01-a712-c14e1f632415 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:34.742463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6b0b62e5-68a7-4e01-a712-c14e1f632415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:34.780219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af384452-c98c-420b-9592-6655ab363579 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.816018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.816855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.817227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.817595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.823165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 469/947] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:34.823346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:34.823346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.823553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0b62e5-68a7-4e01-a712-c14e1f632415 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.823637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.849328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 479/948] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:34.849328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.876558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.878354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:34.935540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0b62e5-68a7-4e01-a712-c14e1f632415 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.957834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf08472-d50d-4288-baed-06ea180421e2 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:34.971574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.972510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.972883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.973624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.974762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] RBAC: Adding query filter params (interface=public, service_id=04ebc1d6f7e241048ac8fb9dc59ffd11, region_id=RegionOne) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:34.975197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.977083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:34.983293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 470/949] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/endpoints?service_id=04ebc1d6f7e241048ac8fb9dc59ffd11&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:34.984223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:34.994725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:34.995733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:34.996156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:34.996581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:34.997224 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:34.998632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:35.010724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:35.012433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:35.013452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 480/950] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:34 2022] GET /identity/v3/services/compute => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:35.015007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:35.099244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb234fd-dc10-40fb-b07c-8af394445743 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:35.109266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:35.111241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:35.122692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea355c7-6157-4d5f-abb0-b5415c68e2e6 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:35.137483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:35.138168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:35.138447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:35.138722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/b211db19066a4767b09f1fcae194085a/roles/f82ad6f8506b4ef089846400682f59f2` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:35.140965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=b211db19066a4767b09f1fcae194085a, role_id=f82ad6f8506b4ef089846400682f59f2)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:35.142422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:35.156745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 471/951] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:58:34 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/b211db19066a4767b09f1fcae194085a/roles/f82ad6f8506b4ef089846400682f59f2 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:58:35.157572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:35.158968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:35.159855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:35.160342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:35.160703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:35.161465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:35.163010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:35.176507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 481/952] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:35 2022] GET /identity/v3/services => generated 1497 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 02:58:35.288166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb27359d-6787-45f9-ab88-64702b1d27fd None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:35.330057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f62e03-6ac8-494b-8841-b169ae8c98c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:35.330836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f62e03-6ac8-494b-8841-b169ae8c98c2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:35.331141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f62e03-6ac8-494b-8841-b169ae8c98c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:35.331463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f62e03-6ac8-494b-8841-b169ae8c98c2 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:35.332286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f62e03-6ac8-494b-8841-b169ae8c98c2 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:35.332591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f62e03-6ac8-494b-8841-b169ae8c98c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:35.334067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f62e03-6ac8-494b-8841-b169ae8c98c2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:35.340333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 472/953] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:35 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:35.346490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:35.460213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a034d07a-f35f-4f5b-808c-9c2e280eb920 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:35.495879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ef003-e51b-418a-855f-df2b50347e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:35.496859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ef003-e51b-418a-855f-df2b50347e42 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:35.497336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ef003-e51b-418a-855f-df2b50347e42 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:35.497778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ef003-e51b-418a-855f-df2b50347e42 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:35.498682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102ef003-e51b-418a-855f-df2b50347e42 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:35.499191 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102ef003-e51b-418a-855f-df2b50347e42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:35.500730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102ef003-e51b-418a-855f-df2b50347e42 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:35.506198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 482/954] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:35 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:36.617907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbefc61-c10d-4d06-b237-d83ee5fab7aa None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:36.618477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbefc61-c10d-4d06-b237-d83ee5fab7aa None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:36.618477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbefc61-c10d-4d06-b237-d83ee5fab7aa None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:36.620263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 473/955] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:36.628643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bab6b9-0b69-4860-967a-ca9a9da2ab08 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:36.629169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bab6b9-0b69-4860-967a-ca9a9da2ab08 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:36.629169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bab6b9-0b69-4860-967a-ca9a9da2ab08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:36.648268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e1bab6b9-0b69-4860-967a-ca9a9da2ab08 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:36.649472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e1bab6b9-0b69-4860-967a-ca9a9da2ab08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:36.665508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:36.724996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 483/956] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:36 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:36.749245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbefc61-c10d-4d06-b237-d83ee5fab7aa None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:36.781235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:36.858230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbefc61-c10d-4d06-b237-d83ee5fab7aa None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:36.893113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:36.893661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:36.893938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:36.894208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:36.894885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:36.896350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:36.912807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.sql.core [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 26 02:58:36.912807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 26 02:58:36.912807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [parameters: {'user_id': '3e93cae5dad542259bd5c665bf6325b6', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 26 02:58:36.912807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80428) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result = self._query(query) Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application conn.query(q) Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result.read() Jul 26 02:58:36.919869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 26 02:58:36.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application next(self.gen) Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application yield resource Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application next(self.gen) Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.session.rollback() Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.force_reraise() Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise self.value Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application session.commit() Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 26 02:58:36.921901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._prepare_impl() Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.session.flush() Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._flush(objects) Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application compat.raise_( Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application flush_context.execute() Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rec.execute(self) Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 26 02:58:36.923438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 26 02:58:36.924536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result = self._query(query) Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application conn.query(q) Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application result.read() Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 26 02:58:36.925692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [parameters: {'user_id': '3e93cae5dad542259bd5c665bf6325b6', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:36.927794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 26 02:58:36.929606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 26 02:58:36.931799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 26 02:58:36.931799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 26 02:58:36.931799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 26 02:58:36.931799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 26 02:58:36.931799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 26 02:58:36.931799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:36.931799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 474/957] 10.222.0.42 () {60 vars in 1131 bytes} [Tue Jul 26 02:58:36 2022] POST /identity/v3/users => generated 160 bytes in 173 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:36.932697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bab6b9-0b69-4860-967a-ca9a9da2ab08 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:37.039122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bab6b9-0b69-4860-967a-ca9a9da2ab08 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:37.075468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a553bf5c-ef83-4039-b5c4-ebc1ec164e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:37.076492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553bf5c-ef83-4039-b5c4-ebc1ec164e5b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.076970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553bf5c-ef83-4039-b5c4-ebc1ec164e5b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.077439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a553bf5c-ef83-4039-b5c4-ebc1ec164e5b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.085669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a553bf5c-ef83-4039-b5c4-ebc1ec164e5b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:37.087461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a553bf5c-ef83-4039-b5c4-ebc1ec164e5b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a553bf5c-ef83-4039-b5c4-ebc1ec164e5b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:37.100719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:37.101742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 26 02:58:37.102717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:37.105456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 484/958] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:37.111834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:37.227572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:37.242597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3e2919-b2fa-4edd-a827-762631ded670 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:37.244660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51bd61-41a7-4f9a-ba52-5eafb24eaf9f None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.244946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51bd61-41a7-4f9a-ba52-5eafb24eaf9f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.245221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51bd61-41a7-4f9a-ba52-5eafb24eaf9f None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.251031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 485/959] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:37.252313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4a7801-2aed-4e20-9b06-e7812a45cf62 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.252599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4a7801-2aed-4e20-9b06-e7812a45cf62 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.252875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4a7801-2aed-4e20-9b06-e7812a45cf62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.278637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-6f4a7801-2aed-4e20-9b06-e7812a45cf62 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:37.279898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6f4a7801-2aed-4e20-9b06-e7812a45cf62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:37.282193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201c96bb-a68a-447c-a57f-b65f33a8f78f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:37.282868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:37.283944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c96bb-a68a-447c-a57f-b65f33a8f78f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.284445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c96bb-a68a-447c-a57f-b65f33a8f78f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.284954 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201c96bb-a68a-447c-a57f-b65f33a8f78f None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.286089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201c96bb-a68a-447c-a57f-b65f33a8f78f None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:37.286522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201c96bb-a68a-447c-a57f-b65f33a8f78f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:37.288341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201c96bb-a68a-447c-a57f-b65f33a8f78f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:37.336614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 475/960] 10.222.0.42 () {56 vars in 1101 bytes} [Tue Jul 26 02:58:37 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:37.415007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 486/961] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:37 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:37.421204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323160df-9fef-42c3-9763-b30192be3c9c None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.421634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323160df-9fef-42c3-9763-b30192be3c9c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.422011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323160df-9fef-42c3-9763-b30192be3c9c None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.427949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 476/962] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:37.430048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbfaaf8-737d-43c5-a9c3-5e10f9d023ff None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.430422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbfaaf8-737d-43c5-a9c3-5e10f9d023ff None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.430769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbfaaf8-737d-43c5-a9c3-5e10f9d023ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.442605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323160df-9fef-42c3-9763-b30192be3c9c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:37.458567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3bbfaaf8-737d-43c5-a9c3-5e10f9d023ff None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:37.459830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3bbfaaf8-737d-43c5-a9c3-5e10f9d023ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:37.576002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 487/963] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:37 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:37.600726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbfaaf8-737d-43c5-a9c3-5e10f9d023ff None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:37.614692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323160df-9fef-42c3-9763-b30192be3c9c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:37.664393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720373a3-a33a-4028-817f-1be898073ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:37.665041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720373a3-a33a-4028-817f-1be898073ee2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.665322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720373a3-a33a-4028-817f-1be898073ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.665594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720373a3-a33a-4028-817f-1be898073ee2 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.677139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720373a3-a33a-4028-817f-1be898073ee2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-720373a3-a33a-4028-817f-1be898073ee2 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:37.682069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:37.683213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:37.684260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 477/964] 10.222.0.42 () {56 vars in 1120 bytes} [Tue Jul 26 02:58:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:37.685151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0498ff6c-f8a9-47dd-b1d4-1b9ebee4d104 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.685151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0498ff6c-f8a9-47dd-b1d4-1b9ebee4d104 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.685358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0498ff6c-f8a9-47dd-b1d4-1b9ebee4d104 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.686611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 478/965] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:37.691524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0498ff6c-f8a9-47dd-b1d4-1b9ebee4d104 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:37.707925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:37.709925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:37.743720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbfaaf8-737d-43c5-a9c3-5e10f9d023ff None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:37.777418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2ec5c3-3534-471b-bb36-62e2755a2ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:37.778089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2ec5c3-3534-471b-bb36-62e2755a2ea8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.778366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2ec5c3-3534-471b-bb36-62e2755a2ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.778598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2ec5c3-3534-471b-bb36-62e2755a2ea8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.779287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2ec5c3-3534-471b-bb36-62e2755a2ea8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:37.780655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2ec5c3-3534-471b-bb36-62e2755a2ea8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-af2ec5c3-3534-471b-bb36-62e2755a2ea8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:37.789536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:37.790582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:37.797138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 488/966] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:37 2022] GET /identity/v3/services/object-store => generated 93 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:37.797955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f275f7d-01c1-45b6-b9c2-8b90d0b0c43c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.797955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f275f7d-01c1-45b6-b9c2-8b90d0b0c43c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.797955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f275f7d-01c1-45b6-b9c2-8b90d0b0c43c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.798147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0498ff6c-f8a9-47dd-b1d4-1b9ebee4d104 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:37.810380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3f275f7d-01c1-45b6-b9c2-8b90d0b0c43c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:37.811623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3f275f7d-01c1-45b6-b9c2-8b90d0b0c43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:37.831881 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:37.832445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.832723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.832997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.833715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:37.834017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:37.835443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:37.844285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 479/967] 10.222.0.42 () {56 vars in 1115 bytes} [Tue Jul 26 02:58:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:37.845102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:37.878550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 489/968] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:37 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:37.879514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f275f7d-01c1-45b6-b9c2-8b90d0b0c43c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:37.925143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:37.944998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:37.950472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a04f62-45f5-43c6-baee-3c8e1253a4e8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:37.983648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f275f7d-01c1-45b6-b9c2-8b90d0b0c43c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:37.984400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:37.985048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:37.985308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:37.985588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:37.986290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:37.986597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:37.988025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:37.993595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 480/969] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:37 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:37.994399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:38.024341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.024999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.025272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.025541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.035760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.037196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:38.051927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:38.052993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:38.058888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:38.058888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:38.058888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:38.058888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:38.058888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 490/970] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:38.058888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:38.096524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99611d5b-3ac0-4193-881a-fbbb80fc0aa2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:38.130354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.131007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.131287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.131559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.132272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.133649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:38.143269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:38.162983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 481/971] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:37 2022] POST /identity/v3/services => generated 242 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:38.162983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:38.163322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbef066-965e-4872-a28e-44eb574f6507 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:38.182797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:38.213309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.214290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.214621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.214934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.215913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:38.216600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.218013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:38.224663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 491/972] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:38 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:38.230675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:38.274067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e231c3-916b-4568-aab7-1e4e1430e4c7 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:38.312939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.312939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.313233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.313810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.314215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:38.314525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.316775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:38.332798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 482/973] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:38.339680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:38.350424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbffaae3-84c2-43b4-9d5b-2c61f3031b2a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:38.388114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.388742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.389019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.389295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.390003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.391431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:38.411236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:38.414677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 492/974] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:38 2022] POST /identity/v3/endpoints => generated 349 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:58:38.460989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d548829f-98dc-4c18-b7e0-7054ce9cf662 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:38.505953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.506609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.506885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.507188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.515497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.516971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:38.530024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:38.531441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:38.532454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 483/975] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 02:58:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:38.537275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:38.579039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:38.654905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0873ae01-c501-4383-8f68-7621ae28e5c6 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:38.691195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.692102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.692523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.692959 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.693909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:38.694358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.695984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:38.714433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 493/976] 10.222.0.42 () {56 vars in 1149 bytes} [Tue Jul 26 02:58:38 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:38.720567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:38.825716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60747c3b-977e-4124-9fb0-bc35d1a9a893 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:38.839709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:38.861592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:38.862224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:38.862498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:38.862765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:38.931982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:38.933444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:39.038720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 484/977] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:38 2022] GET /identity/v3/auth/tokens => generated 1560 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:39.044758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:39.140504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:39.151190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b586d7-9af7-4934-b582-9ab292e7ea2b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:39.185637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:39.186203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:39.186479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:39.186753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:39.197193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:39.198677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:39.213563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:39.214657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:39.215597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:39.215597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:39.215597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:39.215597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:39.219315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 494/978] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:39.220849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:39.239095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:39.328107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148f87f6-50ea-4232-93c4-bc3a442c4566 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:39.363305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:39.363937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:39.364215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:39.364489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:39.365186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:39.365481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:39.366875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:39.379099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 485/979] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:39.384137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:39.487212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519f80b-a297-4995-8f94-b2bc4b8ac6ab None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:39.521990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:39.522641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:39.522924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:39.523227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:39.530332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:39.531837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:39.545625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:39.546598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:39.547915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:39.551451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 495/980] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:39.552875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:39.643037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:39.670638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6462ae0f-7ffd-4433-9518-cf6df0cad833 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:39.706732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28df68b4-9ece-491f-97ea-8dbb40a795e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:39.707378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28df68b4-9ece-491f-97ea-8dbb40a795e3 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:39.707655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28df68b4-9ece-491f-97ea-8dbb40a795e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:39.707920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28df68b4-9ece-491f-97ea-8dbb40a795e3 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:39.708688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28df68b4-9ece-491f-97ea-8dbb40a795e3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:39.709026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28df68b4-9ece-491f-97ea-8dbb40a795e3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:39.710455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28df68b4-9ece-491f-97ea-8dbb40a795e3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:39.723847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 486/981] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:39.729815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:39.733081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:39.839314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f27c3f-a5c2-4bd4-b757-78415e6c3865 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:39.874621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea9e2e0-ccf1-4a1f-b0fd-ac608595e783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:39.875268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9e2e0-ccf1-4a1f-b0fd-ac608595e783 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:39.875545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9e2e0-ccf1-4a1f-b0fd-ac608595e783 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:39.875964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea9e2e0-ccf1-4a1f-b0fd-ac608595e783 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:39.876746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea9e2e0-ccf1-4a1f-b0fd-ac608595e783 None admin] RBAC: Adding query filter params (role.id=f82ad6f8506b4ef089846400682f59f2, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=b211db19066a4767b09f1fcae194085a) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:39.876915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea9e2e0-ccf1-4a1f-b0fd-ac608595e783 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:39.878361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea9e2e0-ccf1-4a1f-b0fd-ac608595e783 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:39.885278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 496/982] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:39 2022] GET /identity/v3/role_assignments?user.id=b211db19066a4767b09f1fcae194085a&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=f82ad6f8506b4ef089846400682f59f2 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:40.196075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d37ac-30f5-4ddf-97a2-ca0aaa53873e None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.196429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d37ac-30f5-4ddf-97a2-ca0aaa53873e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.196629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d37ac-30f5-4ddf-97a2-ca0aaa53873e None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.198032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 487/983] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:40.203672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cefec1-daa2-4ce6-904c-37987c6865bc None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.204036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cefec1-daa2-4ce6-904c-37987c6865bc None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.204379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cefec1-daa2-4ce6-904c-37987c6865bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.224029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c9cefec1-daa2-4ce6-904c-37987c6865bc None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:40.225245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c9cefec1-daa2-4ce6-904c-37987c6865bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:40.294845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:40.303779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 497/984] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:40 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:40.325734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d37ac-30f5-4ddf-97a2-ca0aaa53873e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:40.438999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d37ac-30f5-4ddf-97a2-ca0aaa53873e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:40.446735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8569054-039c-4dc1-8473-44dc6cddb85f None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.446984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8569054-039c-4dc1-8473-44dc6cddb85f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.447258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8569054-039c-4dc1-8473-44dc6cddb85f None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.448861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 498/985] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:58:40.453867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7154876-b21f-4f99-a156-d5d944f9d43f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.454148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7154876-b21f-4f99-a156-d5d944f9d43f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.454412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7154876-b21f-4f99-a156-d5d944f9d43f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.475032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a7154876-b21f-4f99-a156-d5d944f9d43f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:40.475343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:40.475919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.476197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.476293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a7154876-b21f-4f99-a156-d5d944f9d43f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:40.476467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.477103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:40.478474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:40.480429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:40.489774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:40.490827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:40.491699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 488/986] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/services/compute => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:40.497152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:40.551014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 499/987] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:40 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:40.573431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7154876-b21f-4f99-a156-d5d944f9d43f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:40.607614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8a1947-71c6-44f8-8b03-57fbfd0805d4 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:40.661805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:40.662469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.662747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.663052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.664086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:40.664409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:40.666133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:40.671949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 489/988] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:40.678318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:40.694408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7154876-b21f-4f99-a156-d5d944f9d43f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:40.723804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:40.732620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:40.733261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.733542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.733819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.734454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:40.735867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:40.744713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:40.745821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:40.746787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 500/989] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/services/object-store => generated 93 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:40.749988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:40.752034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:40.794422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0dd7ab-8cef-4b96-b4b6-8421a8b0f643 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:40.828903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:40.829530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.829809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.830079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.830755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] RBAC: Adding query filter params (type=compute) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:40.831096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:40.832473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:40.838091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 490/990] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:40.844171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:40.859283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b011786-adb5-4679-a9eb-d8ebe4c91806 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:40.894340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:40.895264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.895714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.896132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.896968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:40.897418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:40.898973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:40.904691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 501/991] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:40.910477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:40.946638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029995bf-dec6-4c95-97f9-fd6e577b97bc None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:40.969319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:40.973623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:40.983535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:40.984101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:40.984375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:40.984645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:40.985412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] RBAC: Adding query filter params (interface=public, service_id=3c44faccbece42519aef2aed86e10ea6, region_id=RegionOne) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:40.985718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:40.987135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:40.994390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 491/992] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/endpoints?service_id=3c44faccbece42519aef2aed86e10ea6&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:40.998060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:41.028576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31333412-61c4-4731-ae22-24aa28c88920 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:41.065461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:41.066114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:41.066393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:41.066661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:41.067402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:41.067708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:41.069128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:41.074905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 502/993] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:41.080774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:41.112355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4d41e-625b-41a4-a63f-cfec989af2e6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:41.149073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:41.149710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:41.149994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:41.150261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:41.150929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:41.152358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:41.166988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 492/994] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:40 2022] GET /identity/v3/services => generated 1727 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 02:58:41.199537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c72acdd-a386-4c6c-b0a8-f52ad1b05628 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:41.219787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:41.239851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78a02de-2e25-4f80-a29e-d2859572d458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:41.240411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a02de-2e25-4f80-a29e-d2859572d458 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:41.240685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a02de-2e25-4f80-a29e-d2859572d458 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:41.240960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78a02de-2e25-4f80-a29e-d2859572d458 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:41.241744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78a02de-2e25-4f80-a29e-d2859572d458 None admin] RBAC: Adding query filter params (interface=admin, service_id=04ebc1d6f7e241048ac8fb9dc59ffd11, region_id=RegionOne) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:41.242057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78a02de-2e25-4f80-a29e-d2859572d458 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:41.243512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78a02de-2e25-4f80-a29e-d2859572d458 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:41.252897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 503/995] 10.222.0.42 () {56 vars in 1249 bytes} [Tue Jul 26 02:58:41 2022] GET /identity/v3/endpoints?service_id=04ebc1d6f7e241048ac8fb9dc59ffd11&interface=admin®ion_id=RegionOne => generated 193 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:41.259204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:41.321565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:41.371755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc3e579-331e-4709-8cc9-d64092f291b0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:41.407379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9639ee-9dad-4095-9d92-0a352402dc59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:41.408009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9639ee-9dad-4095-9d92-0a352402dc59 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:41.408290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9639ee-9dad-4095-9d92-0a352402dc59 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:41.408563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9639ee-9dad-4095-9d92-0a352402dc59 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:41.409220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9639ee-9dad-4095-9d92-0a352402dc59 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:41.410598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9639ee-9dad-4095-9d92-0a352402dc59 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:41.417463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 493/996] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:41 2022] GET /identity/v3/services => generated 1727 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 02:58:41.965105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07163fbf-88b5-406f-822e-fee257146973 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:41.965671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07163fbf-88b5-406f-822e-fee257146973 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:41.965671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07163fbf-88b5-406f-822e-fee257146973 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:41.967237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 504/997] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:41.972780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425526e9-5b96-4c01-8fac-71d160f50796 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:41.973217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425526e9-5b96-4c01-8fac-71d160f50796 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:41.973363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425526e9-5b96-4c01-8fac-71d160f50796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:41.995617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-425526e9-5b96-4c01-8fac-71d160f50796 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:41.997175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-425526e9-5b96-4c01-8fac-71d160f50796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:42.067277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 494/998] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:41 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:42.089286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07163fbf-88b5-406f-822e-fee257146973 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:42.194642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07163fbf-88b5-406f-822e-fee257146973 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:42.229608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:42.230232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:42.230479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:42.230752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] PATH_INFO: `/v3/services/image` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:42.231734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:42.232806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:42.241892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:42.242885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:42.249088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 505/999] 10.222.0.42 () {56 vars in 1105 bytes} [Tue Jul 26 02:58:42 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:42.249900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425526e9-5b96-4c01-8fac-71d160f50796 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:42.301352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:42.354303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425526e9-5b96-4c01-8fac-71d160f50796 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:42.389654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334bbb91-8e06-4abd-93c7-082413a3e756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:42.390285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334bbb91-8e06-4abd-93c7-082413a3e756 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:42.390564 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334bbb91-8e06-4abd-93c7-082413a3e756 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:42.390836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334bbb91-8e06-4abd-93c7-082413a3e756 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:42.391551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334bbb91-8e06-4abd-93c7-082413a3e756 None admin] RBAC: Adding query filter params (name=image) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:42.391857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334bbb91-8e06-4abd-93c7-082413a3e756 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:42.393246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334bbb91-8e06-4abd-93c7-082413a3e756 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:42.402917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 495/1000] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:42 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:42.404205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:42.453729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:42.510525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eab01b-3b1f-4421-9cda-0ff8f08e54e4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:42.546795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37463db1-5b35-43e2-af2e-3102342d0d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:42.547460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37463db1-5b35-43e2-af2e-3102342d0d8c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:42.547724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37463db1-5b35-43e2-af2e-3102342d0d8c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:42.547990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37463db1-5b35-43e2-af2e-3102342d0d8c None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:42.548671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37463db1-5b35-43e2-af2e-3102342d0d8c None admin] RBAC: Adding query filter params (type=image) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:42.548970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37463db1-5b35-43e2-af2e-3102342d0d8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:42.550337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37463db1-5b35-43e2-af2e-3102342d0d8c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:42.562978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 506/1001] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:42 2022] GET /identity/v3/services?type=image => generated 122 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:43.152296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d86818-8b2e-4d03-ae37-511877b0a32b None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.152843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d86818-8b2e-4d03-ae37-511877b0a32b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.152843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d86818-8b2e-4d03-ae37-511877b0a32b None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.154160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 496/1002] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:43.159655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ea16a-2078-412d-adf1-683a85ee4435 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.160076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ea16a-2078-412d-adf1-683a85ee4435 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.160269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ea16a-2078-412d-adf1-683a85ee4435 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.178778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d21ea16a-2078-412d-adf1-683a85ee4435 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:43.179987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d21ea16a-2078-412d-adf1-683a85ee4435 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:43.249332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 507/1003] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:43 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:43.271337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d86818-8b2e-4d03-ae37-511877b0a32b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:43.377090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d86818-8b2e-4d03-ae37-511877b0a32b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:43.390921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:43.394053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0676ca0-a193-4b71-88ad-e01638650107 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.394420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0676ca0-a193-4b71-88ad-e01638650107 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.394703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0676ca0-a193-4b71-88ad-e01638650107 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.396246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 508/1004] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:43.401660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5fb71-bfbe-4680-a183-be8226b8c798 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.401948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5fb71-bfbe-4680-a183-be8226b8c798 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.402241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5fb71-bfbe-4680-a183-be8226b8c798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.414845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:43.415527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.415803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.416076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.416705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:43.418144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:43.421635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f3d5fb71-bfbe-4680-a183-be8226b8c798 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:43.422906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f3d5fb71-bfbe-4680-a183-be8226b8c798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:43.427230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:43.428201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 26 02:58:43.429057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:43.434358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 497/1005] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:43 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:43.435237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:43.459418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:43.497752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 509/1006] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:43 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:43.519512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5fb71-bfbe-4680-a183-be8226b8c798 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:43.545093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0f93e5-6ab1-4176-9628-545adb0f1ed2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:43.580782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:43.581707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.581986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.582260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.582963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] RBAC: Adding query filter params (name=compute) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:43.583284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:43.584663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:43.589545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 498/1007] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:43 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:43.595973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:43.629442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5fb71-bfbe-4680-a183-be8226b8c798 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:43.641031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:43.668934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:43.669848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.670026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.670267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.671008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:43.672363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:43.682903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:43.683969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:43.684797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 510/1008] 10.222.0.42 () {56 vars in 1126 bytes} [Tue Jul 26 02:58:43 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:43.690328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:43.707502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10b4b54-f7b0-499c-91dc-c6056f8d468a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:43.727446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:43.742748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:43.743410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.743696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.743971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.744662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] RBAC: Adding query filter params (type=compute) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:43.744969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:43.746434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:43.755915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 499/1009] 10.222.0.42 () {56 vars in 1112 bytes} [Tue Jul 26 02:58:43 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:43.756949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:43.803620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b679a8b-a6c1-4d22-ae79-9743912407ce None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:43.839927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:43.840555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.840844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.841131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.841808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:43.842129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:43.843883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:43.849238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 511/1010] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:43 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:43.854874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:43.859949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07dc95f-0dd4-4882-af21-922665c4eb6b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:43.863082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:43.897672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:43.898302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:43.898580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:43.898858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:43.899602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:43.900983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:43.922887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 500/1011] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:43 2022] POST /identity/v3/endpoints => generated 334 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:43.961373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:43.966167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501243fd-fb1d-4793-b75e-1bd2c88d9372 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:44.020690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127ae97e-e999-4392-bd01-cb5d1ec15beb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:44.021330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127ae97e-e999-4392-bd01-cb5d1ec15beb None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:44.021605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127ae97e-e999-4392-bd01-cb5d1ec15beb None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:44.021880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127ae97e-e999-4392-bd01-cb5d1ec15beb None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:44.022588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127ae97e-e999-4392-bd01-cb5d1ec15beb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:44.022896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127ae97e-e999-4392-bd01-cb5d1ec15beb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:44.024377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127ae97e-e999-4392-bd01-cb5d1ec15beb None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:44.030710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 512/1012] 10.222.0.42 () {56 vars in 1122 bytes} [Tue Jul 26 02:58:43 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:44.036623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:44.178160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b681a-65bd-4036-a318-6837fd503b3a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:44.217910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01ed50b-18d8-4b08-b69f-8f88c50a5368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:44.218910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ed50b-18d8-4b08-b69f-8f88c50a5368 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:44.219223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ed50b-18d8-4b08-b69f-8f88c50a5368 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:44.219501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01ed50b-18d8-4b08-b69f-8f88c50a5368 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:44.220248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01ed50b-18d8-4b08-b69f-8f88c50a5368 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:44.221673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01ed50b-18d8-4b08-b69f-8f88c50a5368 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:44.241151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 501/1013] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:44 2022] POST /identity/v3/endpoints => generated 325 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:44.590723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1b6210-fe6b-4741-996c-adb88e999110 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:44.591252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1b6210-fe6b-4741-996c-adb88e999110 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:44.591252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1b6210-fe6b-4741-996c-adb88e999110 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:44.592849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 513/1014] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:44.598461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4edf81-a2ad-4751-b890-5bcdf6f62b23 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:44.598754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4edf81-a2ad-4751-b890-5bcdf6f62b23 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:44.599166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4edf81-a2ad-4751-b890-5bcdf6f62b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:44.617651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3d4edf81-a2ad-4751-b890-5bcdf6f62b23 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:44.619216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3d4edf81-a2ad-4751-b890-5bcdf6f62b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:44.635447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:44.698996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 502/1015] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:44 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:44.720734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1b6210-fe6b-4741-996c-adb88e999110 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:44.827932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1b6210-fe6b-4741-996c-adb88e999110 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:44.863094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6736015f-e487-4b16-97fb-7e983f9757b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:44.863720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6736015f-e487-4b16-97fb-7e983f9757b4 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:44.864001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6736015f-e487-4b16-97fb-7e983f9757b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:44.864280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6736015f-e487-4b16-97fb-7e983f9757b4 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:44.864982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6736015f-e487-4b16-97fb-7e983f9757b4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:44.866388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6736015f-e487-4b16-97fb-7e983f9757b4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:44.890975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 514/1016] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:44 2022] POST /identity/v3/services => generated 242 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:46.081971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b68f5f-4ecc-4ed8-a0c9-84b961802055 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.082511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b68f5f-4ecc-4ed8-a0c9-84b961802055 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.082511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b68f5f-4ecc-4ed8-a0c9-84b961802055 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.083934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 503/1017] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:46.089855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc387bf6-68f9-4f21-81ce-31e396d001d5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.090386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc387bf6-68f9-4f21-81ce-31e396d001d5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.090612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc387bf6-68f9-4f21-81ce-31e396d001d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.110235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-fc387bf6-68f9-4f21-81ce-31e396d001d5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:46.111461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fc387bf6-68f9-4f21-81ce-31e396d001d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:46.126407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:46.202997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 515/1018] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:46 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:46.228472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b68f5f-4ecc-4ed8-a0c9-84b961802055 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:46.267619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ebd8ed-dff6-49ef-a748-4de351fdf6ba None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.267818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ebd8ed-dff6-49ef-a748-4de351fdf6ba None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.268051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ebd8ed-dff6-49ef-a748-4de351fdf6ba None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.269415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 516/1019] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:58:46.274525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b7fdd-61b9-4502-91a5-160a95a04469 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.274808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b7fdd-61b9-4502-91a5-160a95a04469 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.275105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b7fdd-61b9-4502-91a5-160a95a04469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.293390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-072b7fdd-61b9-4502-91a5-160a95a04469 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:46.294568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-072b7fdd-61b9-4502-91a5-160a95a04469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:46.336713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b68f5f-4ecc-4ed8-a0c9-84b961802055 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:46.373679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:46.374335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.374608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.374878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.375279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 517/1020] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:46 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 02:58:46.385552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:46.386048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b7fdd-61b9-4502-91a5-160a95a04469 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:46.390451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:46.391426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:46.396176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 504/1021] 10.222.0.42 () {56 vars in 1120 bytes} [Tue Jul 26 02:58:46 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:46.397549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:46.407534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:46.470633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:46.496702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b7fdd-61b9-4502-91a5-160a95a04469 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:46.508047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503fb6c-2347-4259-adad-2682a304981c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:46.532459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:46.533037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.533316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.533724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.534982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:46.536683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:46.538510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 518/1022] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:46.545883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:46.548189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:46.548954 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.549231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.549610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.550339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:46.550639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:46.552071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:46.566769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 505/1023] 10.222.0.42 () {56 vars in 1115 bytes} [Tue Jul 26 02:58:46 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:46.568709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:46.637816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:46.660185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2950f5b2-4c25-46f5-a9f5-65bd22024073 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:46.679500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30fc72a-52f9-47f3-bc53-066b0df5fe65 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:46.696066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:46.696699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.696983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.697260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.697949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:46.699476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:46.714302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:46.718758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3755096-25c0-4ad5-b2df-2cdb6d51025b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:46.719513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3755096-25c0-4ad5-b2df-2cdb6d51025b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.719855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3755096-25c0-4ad5-b2df-2cdb6d51025b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.720226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3755096-25c0-4ad5-b2df-2cdb6d51025b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.728525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 519/1024] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:58:46 2022] POST /identity/v3/projects => generated 311 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:46.742680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3755096-25c0-4ad5-b2df-2cdb6d51025b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:46.744365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3755096-25c0-4ad5-b2df-2cdb6d51025b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e3755096-25c0-4ad5-b2df-2cdb6d51025b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:46.762400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:46.763433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:46.764393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:46.764393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:46.764393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:46.764393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:46.764393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 506/1025] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:46.769821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:46.884292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8050b1d3-88c2-4b37-8b07-6b8420887d83 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:46.901078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd0799-e215-4fd3-850b-62661f764b98 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.901690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd0799-e215-4fd3-850b-62661f764b98 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.902146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd0799-e215-4fd3-850b-62661f764b98 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.903737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 507/1026] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:46.910179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b047712-c594-40c5-bafc-c1f040785e7e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.910676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b047712-c594-40c5-bafc-c1f040785e7e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.911005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b047712-c594-40c5-bafc-c1f040785e7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.926631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:46.927294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:46.927581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:46.927856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:46.928564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:46.928868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:46.930247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:46.930552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2b047712-c594-40c5-bafc-c1f040785e7e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:46.931787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2b047712-c594-40c5-bafc-c1f040785e7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:46.942513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 520/1027] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:46.948442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.015531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.015780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 508/1028] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:46 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:47.035218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b047712-c594-40c5-bafc-c1f040785e7e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.061294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215053e4-7a25-411f-b7cb-a98815c45125 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.085430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.104697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.105411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.106044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.106044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.113870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.115377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:47.130539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:47.131628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:47.133179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 521/1029] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:58:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:47.137923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.145039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b047712-c594-40c5-bafc-c1f040785e7e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.182282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.182997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.183280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.183560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] PATH_INFO: `/v3/services/image` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.184202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.185591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:47.194612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:47.195638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:47.196626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 509/1030] 10.222.0.42 () {56 vars in 1105 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/services/image => generated 86 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:47.202045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.256575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82e357a-ded8-46c1-b5f3-9616da02d6d1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.271437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.294049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.294682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.294993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.295278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.296091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:47.296404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.297849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.317725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 522/1031] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:47.319514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade04604-2274-4a57-9e36-1028820db700 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.323654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.337561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.362225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.362992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.363284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.363560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.364252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] RBAC: Adding query filter params (name=image) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:47.364965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.366052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.376522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 510/1032] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/services?name=image => generated 122 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:47.378063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.424556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa59e647-7b8e-42e3-8cfd-c31e833d4d42 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.464863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.465491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.465771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.466044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.490850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f695b25-9f5b-4c01-aba9-b418175e5849 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.507584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.530039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.531144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.531760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.532102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.532832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] RBAC: Adding query filter params (type=image) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:47.533112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.534155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.534498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.535784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.540629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 511/1033] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/services?type=image => generated 350 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:47.546451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.591825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.645100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 523/1034] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 26 02:58:47.650795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.660267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44e7f94-1170-4f3e-bc44-f98a87ba6224 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.697984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.698906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.698906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.699149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] PATH_INFO: `/v3/endpoints` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.699914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] RBAC: Adding query filter params (interface=public, service_id=858095fa92f44e7fbd9b0bf439426378, region_id=RegionOne) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:47.700229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.701666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.711274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 512/1035] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/endpoints?service_id=858095fa92f44e7fbd9b0bf439426378&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:47.712476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.725605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.755735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36567308-69a8-4286-8823-cda7b39e0773 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.791454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.792505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.792954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.793236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.804176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.805643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.819753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d53652-abbe-4d9b-9dd4-c108d3ea9bee None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:47.823244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:47.824182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:47.825080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:47.825080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:47.825080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:47.825080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:47.825080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 524/1036] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:47.829042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.830162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:47.863309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.863904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.864182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.864451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.865112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.866833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:47.883013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 513/1037] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/services => generated 1957 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 02:58:47.952613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25904f66-dcef-4e4c-8860-9f9f7b938e73 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:47.978639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:47.994547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:47.994547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:47.994547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:47.994547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:47.994934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:47.994934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:47.995421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:48.011763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 525/1038] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:48.015745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:48.122658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e313bc5-4ecc-4138-a6e4-4ae9de1cbae9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:48.156052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:48.156773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:48.157051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:48.157325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:48.164027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:48.165468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:48.176950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:48.177868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:48.178753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 514/1039] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:48.184211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:48.289734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8250a5-0bf6-4c1a-9f32-8d0c58680469 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:48.324421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda18f7d-8919-4836-88c5-413ad1413afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:48.325056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda18f7d-8919-4836-88c5-413ad1413afe None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:48.325334 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda18f7d-8919-4836-88c5-413ad1413afe None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:48.325611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda18f7d-8919-4836-88c5-413ad1413afe None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:48.326381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda18f7d-8919-4836-88c5-413ad1413afe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:48.326694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda18f7d-8919-4836-88c5-413ad1413afe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:48.328184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda18f7d-8919-4836-88c5-413ad1413afe None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:48.340506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 526/1040] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:48.346919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:48.381441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:48.456807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23c13c-d8c6-411d-b38b-f1b335b6ba90 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:48.493681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962ea01a-54b2-43ad-bd86-e146eb22495b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:48.494538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962ea01a-54b2-43ad-bd86-e146eb22495b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:48.494851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962ea01a-54b2-43ad-bd86-e146eb22495b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:48.495253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962ea01a-54b2-43ad-bd86-e146eb22495b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:48.496213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962ea01a-54b2-43ad-bd86-e146eb22495b None admin] RBAC: Adding query filter params (role.id=f82ad6f8506b4ef089846400682f59f2, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=e2c09978fa3f44d895cbc17efdca2b32) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:48.496529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962ea01a-54b2-43ad-bd86-e146eb22495b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:48.498029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962ea01a-54b2-43ad-bd86-e146eb22495b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:48.505208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 515/1041] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:48 2022] GET /identity/v3/role_assignments?user.id=e2c09978fa3f44d895cbc17efdca2b32&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=f82ad6f8506b4ef089846400682f59f2 => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:48.789126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4e3ab5-3a53-4ffd-a6c0-0126c528768f None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:48.789400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4e3ab5-3a53-4ffd-a6c0-0126c528768f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:48.789681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4e3ab5-3a53-4ffd-a6c0-0126c528768f None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:48.791225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 527/1042] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:48.796880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc5221-f10e-4f1e-9c45-99fe43de572c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:48.797125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc5221-f10e-4f1e-9c45-99fe43de572c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:48.797434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc5221-f10e-4f1e-9c45-99fe43de572c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:48.817919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-65dc5221-f10e-4f1e-9c45-99fe43de572c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:48.819989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-65dc5221-f10e-4f1e-9c45-99fe43de572c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:48.895823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 516/1043] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:48 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:48.920193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4e3ab5-3a53-4ffd-a6c0-0126c528768f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:48.971541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:49.043165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4e3ab5-3a53-4ffd-a6c0-0126c528768f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:49.081366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46f5d1a-1103-4cb9-b75b-2973827c77a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:49.082019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f5d1a-1103-4cb9-b75b-2973827c77a7 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:49.082296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f5d1a-1103-4cb9-b75b-2973827c77a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:49.082569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46f5d1a-1103-4cb9-b75b-2973827c77a7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:49.083790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46f5d1a-1103-4cb9-b75b-2973827c77a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:49.085243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46f5d1a-1103-4cb9-b75b-2973827c77a7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:49.087036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 528/1044] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:49.092980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc5221-f10e-4f1e-9c45-99fe43de572c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:49.120592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:49.209576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc5221-f10e-4f1e-9c45-99fe43de572c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:49.245616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e32980-7922-4d15-a5ff-c6d54a76583f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:49.246260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e32980-7922-4d15-a5ff-c6d54a76583f None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:49.246539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e32980-7922-4d15-a5ff-c6d54a76583f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:49.246818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e32980-7922-4d15-a5ff-c6d54a76583f None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:49.247544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e32980-7922-4d15-a5ff-c6d54a76583f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:49.248992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e32980-7922-4d15-a5ff-c6d54a76583f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:49.272947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 517/1045] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:58:49 2022] POST /identity/v3/users => generated 285 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:49.916282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b710c21-da9d-415a-a66b-ac3a994172be None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:49.916829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b710c21-da9d-415a-a66b-ac3a994172be None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:49.916829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b710c21-da9d-415a-a66b-ac3a994172be None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:49.918446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 529/1046] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:49.923596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4752c26c-b1f2-42c2-ae92-724154c9183d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:49.923850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4752c26c-b1f2-42c2-ae92-724154c9183d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:49.924157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4752c26c-b1f2-42c2-ae92-724154c9183d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:49.942769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4752c26c-b1f2-42c2-ae92-724154c9183d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:49.944036 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4752c26c-b1f2-42c2-ae92-724154c9183d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:50.022991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 518/1047] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:49 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:50.043411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b710c21-da9d-415a-a66b-ac3a994172be None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:50.152942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b710c21-da9d-415a-a66b-ac3a994172be None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:50.167062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:50.192255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:50.192909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.193197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.193472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] PATH_INFO: `/v3/services/image` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.194108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:50.195560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:50.205564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:50.206615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 26 02:58:50.208116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:50.211410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 530/1048] 10.222.0.42 () {56 vars in 1105 bytes} [Tue Jul 26 02:58:50 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:50.212598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4752c26c-b1f2-42c2-ae92-724154c9183d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:50.225012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:50.318483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4752c26c-b1f2-42c2-ae92-724154c9183d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:50.353082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8172e905-6056-4213-a022-fb0fc09a4249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:50.353711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8172e905-6056-4213-a022-fb0fc09a4249 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.353987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8172e905-6056-4213-a022-fb0fc09a4249 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.354258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8172e905-6056-4213-a022-fb0fc09a4249 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.354983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8172e905-6056-4213-a022-fb0fc09a4249 None admin] RBAC: Adding query filter params (name=image) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:50.355317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8172e905-6056-4213-a022-fb0fc09a4249 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:50.356682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8172e905-6056-4213-a022-fb0fc09a4249 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:50.366526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 519/1049] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:50 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:50.368401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:50.473726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be9dc27-1ac8-4851-878a-5685903ea280 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:50.508346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:50.509022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.509302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.509578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.510265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] RBAC: Adding query filter params (type=image) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:50.510589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:50.510793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747f826-b579-41f4-862e-573f5c61f247 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.511039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747f826-b579-41f4-862e-573f5c61f247 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.511319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747f826-b579-41f4-862e-573f5c61f247 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.512080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:50.512895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 520/1050] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:50.517902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926e7e9-e1e4-4679-84db-871a48b06554 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.518189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926e7e9-e1e4-4679-84db-871a48b06554 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.518701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c926e7e9-e1e4-4679-84db-871a48b06554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.518808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 531/1051] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:50 2022] GET /identity/v3/services?type=image => generated 350 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:50.524511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:50.539108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c926e7e9-e1e4-4679-84db-871a48b06554 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:50.540332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c926e7e9-e1e4-4679-84db-871a48b06554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:50.573996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:50.621730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 521/1052] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:50 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:50.637224 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497fc4cd-39d7-4f9f-b7e0-8b1f052d3ce2 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:50.644005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c926e7e9-e1e4-4679-84db-871a48b06554 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:50.648137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:50.675176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:50.675941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.676220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.676578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.677561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:50.679185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:50.711167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 532/1053] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 02:58:50 2022] POST /identity/v3/endpoints => generated 327 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:58:50.759774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c926e7e9-e1e4-4679-84db-871a48b06554 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:50.807261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:50.808312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.808741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.809162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.821954 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:50.827524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:50.828592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:50.829477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:50.831125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 522/1054] 10.222.0.42 () {56 vars in 1120 bytes} [Tue Jul 26 02:58:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:50.835821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:50.932808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:50.940502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec38f392-300c-4aee-aefb-685c858ff3b2 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:50.975815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:50.976446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:50.976726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:50.976996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:50.977717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:50.978100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:50.979642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:50.988173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 533/1055] 10.222.0.42 () {56 vars in 1115 bytes} [Tue Jul 26 02:58:50 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:50.997714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.068167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:51.114845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4f451f-0f68-45e7-926d-4ab8974197de None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.150485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb68365-602a-4b0f-af59-44d27902d68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.151164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb68365-602a-4b0f-af59-44d27902d68b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.151540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb68365-602a-4b0f-af59-44d27902d68b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.151815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb68365-602a-4b0f-af59-44d27902d68b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.162718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb68365-602a-4b0f-af59-44d27902d68b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:51.164248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb68365-602a-4b0f-af59-44d27902d68b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-7fb68365-602a-4b0f-af59-44d27902d68b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:51.179938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:51.180919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:51.181828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:51.181828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:51.181828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:51.181828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:51.181828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 523/1056] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:51.188140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.293570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98812c1-0b0b-4c1b-bab5-d5741e2a0a9b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.308909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b2e56-696c-4c1f-b266-eacdf3a7b345 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.309419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b2e56-696c-4c1f-b266-eacdf3a7b345 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.309530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b2e56-696c-4c1f-b266-eacdf3a7b345 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.311787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 524/1057] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:51.316204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a67a6-3745-48e3-80b6-c833ff2b4b9a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.316704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a67a6-3745-48e3-80b6-c833ff2b4b9a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.317040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a67a6-3745-48e3-80b6-c833ff2b4b9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.333147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.333773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.334053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.334327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.335058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:51.335370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:51.335509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-261a67a6-3745-48e3-80b6-c833ff2b4b9a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:51.336616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-261a67a6-3745-48e3-80b6-c833ff2b4b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:51.336783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:51.349756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 534/1058] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:51.355299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.375113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:51.410462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 525/1059] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:51 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:51.433374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261a67a6-3745-48e3-80b6-c833ff2b4b9a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.467093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680f4049-6d25-4398-bf59-c046788d191c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.478757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:51.512802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.513626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.513908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.514183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] PATH_INFO: `/v3/users/nova` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.523814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:51.525128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:51.538976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:51.540012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:51.540973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 535/1060] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:51.543387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261a67a6-3745-48e3-80b6-c833ff2b4b9a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.546034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.582972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.583627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.583974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.584251 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.595053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:51.599903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:51.600829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:51.601688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 526/1061] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:51.607444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.636595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:51.660788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78759b5b-f20b-4da2-aae3-7fc50f685f6c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.699278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.699928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.700208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.700486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.701296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:51.701605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:51.703070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:51.714581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1bbab3-fde9-4cab-92d7-6e3c85fcb781 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.716178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:51.727195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 536/1062] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:51.728283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.755129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.755766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.756044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.756324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.757046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:51.757352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:51.758739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:51.771989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 527/1063] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:51.773047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.835999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8209cf2-cd84-47d3-a8f1-137b2582a578 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.874353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.875041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.875326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.875609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.878622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:51.880663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93961862-7e72-4de6-9e23-78540984d142 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:51.889378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:51.891233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:51.906902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:51.907846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:51.908765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:51.908765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:51.908765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:51.908765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:51.908765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 537/1064] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:51.913649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.918431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:51.919134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:51.919417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:51.919692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] PATH_INFO: `/v3/users/f051d1f037a9450fae743b092291d152` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:51.935005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] RBAC: Authorizing `identity:get_user(user_id=f051d1f037a9450fae743b092291d152)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:51.936894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:51.938643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 528/1065] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/users/f051d1f037a9450fae743b092291d152 => generated 285 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:51.944294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:51.969726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:52.026965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7a5353-1e80-4f12-be35-67ef37b876aa None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:52.054926 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6cfa8c-f165-4252-91d6-7b5a579a3b63 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:52.061233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:52.061855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.062134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.062406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.063217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:52.063519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:52.064898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:52.077226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 538/1066] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:52.082843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:52.095917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:52.096557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.096747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.097022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.137138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:52.167538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:52.169019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:52.189129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aa7699-21db-446c-8a9e-6287e064048d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:52.195187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:52.225877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:52.226524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.226910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.227213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.234629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:52.236127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:52.248127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:52.249236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:52.254995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 539/1067] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:52.255503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:52.283170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 529/1068] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:51 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:52.289026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:52.359592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72347f-6063-4ab8-9f0f-620ab82369ae None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:52.369967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:52.398242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:52.398870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.399176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.399450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.400214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:52.400521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:52.400616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d995863-40f0-4a8b-949a-748dd1acabfc None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:52.402196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:52.415072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 540/1069] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:52.421074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:52.431582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:52.445236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:52.445839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.446122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.446405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.447832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] RBAC: Authorizing `identity:get_project(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:52.449088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:52.451112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 530/1070] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:58:52 2022] GET /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4 => generated 311 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:52.456884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:52.529967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25aa4e50-e0e7-48d4-b357-66f29455b604 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:52.560873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b877d4dc-a1de-49c7-878e-a2cb4b1657c9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:52.566180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a4c847-5c67-4c8a-899d-2764948fd088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:52.566864 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4c847-5c67-4c8a-899d-2764948fd088 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.567171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4c847-5c67-4c8a-899d-2764948fd088 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.567445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4c847-5c67-4c8a-899d-2764948fd088 None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/e2c09978fa3f44d895cbc17efdca2b32/roles/f82ad6f8506b4ef089846400682f59f2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.570034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a4c847-5c67-4c8a-899d-2764948fd088 None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=e2c09978fa3f44d895cbc17efdca2b32, role_id=f82ad6f8506b4ef089846400682f59f2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:52.571873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a4c847-5c67-4c8a-899d-2764948fd088 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:52.587840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 541/1071] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:58:52 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/e2c09978fa3f44d895cbc17efdca2b32/roles/f82ad6f8506b4ef089846400682f59f2 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 02:58:52.603876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4291c20-df8b-4a81-a594-641943d7cebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:52.605007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4291c20-df8b-4a81-a594-641943d7cebe None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.605515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4291c20-df8b-4a81-a594-641943d7cebe None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.605980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4291c20-df8b-4a81-a594-641943d7cebe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.607026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4291c20-df8b-4a81-a594-641943d7cebe None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4, user.id=f051d1f037a9450fae743b092291d152) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:52.607597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4291c20-df8b-4a81-a594-641943d7cebe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:52.609441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4291c20-df8b-4a81-a594-641943d7cebe None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:52.616768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 531/1072] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:52 2022] GET /identity/v3/role_assignments?user.id=f051d1f037a9450fae743b092291d152&scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:52.756814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc23de-8566-496c-86b6-ee2cdcfdb28b None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.757183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc23de-8566-496c-86b6-ee2cdcfdb28b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.757361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc23de-8566-496c-86b6-ee2cdcfdb28b None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.759032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 542/1073] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:52.765261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ace029-6f1d-4b53-a1bc-a8d0b8db1390 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:52.765894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ace029-6f1d-4b53-a1bc-a8d0b8db1390 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:52.766228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ace029-6f1d-4b53-a1bc-a8d0b8db1390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:52.789340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-73ace029-6f1d-4b53-a1bc-a8d0b8db1390 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:52.790551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-73ace029-6f1d-4b53-a1bc-a8d0b8db1390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:52.846064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:52.865970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 532/1074] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:52.888794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc23de-8566-496c-86b6-ee2cdcfdb28b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:52.921235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:53.005579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc23de-8566-496c-86b6-ee2cdcfdb28b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:53.041936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:53.042576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:53.042816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:53.043157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:53.119411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:53.120869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:53.148257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:53.228234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 543/1075] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:52 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:53.229727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ace029-6f1d-4b53-a1bc-a8d0b8db1390 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:53.332177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ace029-6f1d-4b53-a1bc-a8d0b8db1390 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:53.368902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec8cd58-c529-46fd-be2b-74b8a09352da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:53.369552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec8cd58-c529-46fd-be2b-74b8a09352da None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:53.369836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec8cd58-c529-46fd-be2b-74b8a09352da None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:53.370244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec8cd58-c529-46fd-be2b-74b8a09352da None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:53.380324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec8cd58-c529-46fd-be2b-74b8a09352da None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:53.381777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec8cd58-c529-46fd-be2b-74b8a09352da None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-8ec8cd58-c529-46fd-be2b-74b8a09352da None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:53.396509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:53.397440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:53.404163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:53.404163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:53.404163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:53.404163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:53.404163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 533/1076] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:53.404463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:53.502431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3cdf75-e0ab-4f47-9233-055effd85020 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:53.528716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:53.538142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f5aa10-4da5-49f8-9d80-e385b340b8a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:53.538767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5aa10-4da5-49f8-9d80-e385b340b8a4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:53.539088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5aa10-4da5-49f8-9d80-e385b340b8a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:53.539359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5aa10-4da5-49f8-9d80-e385b340b8a4 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:53.540041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f5aa10-4da5-49f8-9d80-e385b340b8a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:53.540338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f5aa10-4da5-49f8-9d80-e385b340b8a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:53.541701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f5aa10-4da5-49f8-9d80-e385b340b8a4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:53.562974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 544/1077] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:54.601828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aad74c-6e42-406c-9354-b8cf781509b3 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:54.602356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aad74c-6e42-406c-9354-b8cf781509b3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:54.602356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aad74c-6e42-406c-9354-b8cf781509b3 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:54.603790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 534/1078] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:54.609233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c5d5ff-b570-48d6-b39b-b876786a5d5e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:54.609672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c5d5ff-b570-48d6-b39b-b876786a5d5e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:54.609755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c5d5ff-b570-48d6-b39b-b876786a5d5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:54.627439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d76ddb0-310d-4228-ab35-8ba61137241b None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:54.627828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d76ddb0-310d-4228-ab35-8ba61137241b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:54.628104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d76ddb0-310d-4228-ab35-8ba61137241b None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:54.629129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-23c5d5ff-b570-48d6-b39b-b876786a5d5e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:54.629475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 535/1079] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:54.630341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-23c5d5ff-b570-48d6-b39b-b876786a5d5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:54.634983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd26a3-9372-44a4-b1dd-5e19018d53e0 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:54.635249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd26a3-9372-44a4-b1dd-5e19018d53e0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:54.635526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd26a3-9372-44a4-b1dd-5e19018d53e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:54.654660 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-5efd26a3-9372-44a4-b1dd-5e19018d53e0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:54.655909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5efd26a3-9372-44a4-b1dd-5e19018d53e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:54.688201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:54.700117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 545/1080] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:54 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:54.723637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c5d5ff-b570-48d6-b39b-b876786a5d5e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:54.731729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 536/1081] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:54 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:54.753816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd26a3-9372-44a4-b1dd-5e19018d53e0 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:54.820276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c5d5ff-b570-48d6-b39b-b876786a5d5e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:54.837994 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:54.854986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd26a3-9372-44a4-b1dd-5e19018d53e0 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:54.855598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:54.856241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:54.856524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:54.856795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:54.866634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:54.871586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:54.872553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:54.873790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 546/1082] 10.222.0.42 () {56 vars in 1120 bytes} [Tue Jul 26 02:58:54 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:54.879001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:54.891597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:54.892253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:54.892529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:54.892799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:54.902864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:54.907689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:54.908656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:54.909647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 537/1083] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:54.914916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:54.942518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:54.981513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56df4a1b-7acc-49c5-920b-9940bc25cafd None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.015961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.016599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.016883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.017159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.017888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:55.018191 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.019576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daf7759-1a0d-4ed8-b213-6e0e10de08b9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.019686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.028416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 547/1084] 10.222.0.42 () {56 vars in 1115 bytes} [Tue Jul 26 02:58:54 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:55.034273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.053350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.053989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.054269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.054541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.055295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:55.055602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.056990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.065403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 538/1085] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:55.071324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.074600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:55.133362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6eba767-ed93-4de8-8584-4722584ffa0e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.158124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:55.166347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.167086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.167375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.167646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.170834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04c77f6-9c62-4fca-b85b-7db95a64cfa2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.178832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.180326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:55.194258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:55.195221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:55.196136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:55.196136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:55.196136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:55.196136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:55.196136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 548/1086] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:55.201342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.210184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.210840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.211151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.211412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] PATH_INFO: `/v3/users/f051d1f037a9450fae743b092291d152` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.212691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] RBAC: Authorizing `identity:get_user(user_id=f051d1f037a9450fae743b092291d152)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.214140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.221008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 539/1087] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:58:55 2022] GET /identity/v3/users/f051d1f037a9450fae743b092291d152 => generated 285 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:55.221957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.291470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:55.300456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1d335-9d47-43ff-8624-a000fa22360b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.319325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a38dcc-b55e-4b33-9458-98c8f45dc351 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.333355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.333976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.334254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.334529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.335272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:55.335573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.336945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.349078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 549/1088] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:55.354983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.358552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.359226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.359588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.359900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.361880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] RBAC: Authorizing `identity:get_project(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.363347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.365028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 540/1089] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:58:55 2022] GET /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:58:55.371004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.396284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:55.455677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a90abb6-7385-4e42-89d6-e394bf9d86e8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.472143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0a594e-a7cb-46ec-993d-eee341cf45ca None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.490347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.491024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.491306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.491579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] PATH_INFO: `/v3/users/nova` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.498878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.500376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.506589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.507304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.507581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.507856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4/users/f051d1f037a9450fae743b092291d152/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.509818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] RBAC: Authorizing `identity:create_grant(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4, user_id=f051d1f037a9450fae743b092291d152, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.511318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 02:58:55.512339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:55.513242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:55.514203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 550/1090] 10.222.0.42 () {56 vars in 1093 bytes} [Tue Jul 26 02:58:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 02:58:55.519758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.527521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 541/1091] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:58:55 2022] PUT /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4/users/f051d1f037a9450fae743b092291d152/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 02:58:55.535986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:55.643489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab04b9a-3a61-4603-b6e9-6b2d38cac7fb None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.683651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.684214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.684490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.684755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.685853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:55.685853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.687881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:55.706781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 551/1092] 10.222.0.42 () {56 vars in 1133 bytes} [Tue Jul 26 02:58:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:55.712843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:55.801627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:55.820418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d11c9-a550-447d-9749-23d5ba3f25ef None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:55.857076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:55.857703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:55.857981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:55.858255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:55.929444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:55.930904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:56.018460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:56.036913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 542/1093] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:55 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:56.043097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:56.114900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:56.145622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c692c1-577a-44b2-a238-028d9719ae4d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:56.180588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:56.181234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:56.181516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:56.181786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:56.192340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:56.193796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:58:56.209710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:56.210689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:56.211645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 02:58:56.211645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:58:56.211645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 02:58:56.211645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:56.211645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 552/1094] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:58:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:56.217684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:56.324237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23435b-5a8c-4e35-9086-3db393694ab1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:56.359263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:56.359869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:56.360148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:56.360421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:56.361116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:56.361419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:56.362798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:56.374007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 543/1095] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 02:58:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:56.379919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:56.483326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbae67d-c787-4e54-989d-52f248f09bb8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:56.505844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:56.518769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:56.519469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:56.519759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:56.520033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:56.527046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:56.528501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 02:58:56.539829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:56.540822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:58:56.546606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 553/1096] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 02:58:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:56.547142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:56.597441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:56.654427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7097f630-d4ea-46db-97b1-bb41d53dbf15 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:56.690036 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22d099f-029d-48ba-b724-db583f2fdad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:56.690664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22d099f-029d-48ba-b724-db583f2fdad8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:56.690968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22d099f-029d-48ba-b724-db583f2fdad8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:56.691243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22d099f-029d-48ba-b724-db583f2fdad8 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:56.692005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22d099f-029d-48ba-b724-db583f2fdad8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:56.692313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22d099f-029d-48ba-b724-db583f2fdad8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:56.693805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22d099f-029d-48ba-b724-db583f2fdad8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:56.706224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 544/1097] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 02:58:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:56.712486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:56.817030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fed350-fee7-401a-90af-cf11c2db2148 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:56.851353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5389e2e7-e2e6-4710-900b-0b659da25749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:56.851902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389e2e7-e2e6-4710-900b-0b659da25749 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:56.852179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389e2e7-e2e6-4710-900b-0b659da25749 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:56.852462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389e2e7-e2e6-4710-900b-0b659da25749 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:56.853245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5389e2e7-e2e6-4710-900b-0b659da25749 None admin] RBAC: Adding query filter params (role.id=f82ad6f8506b4ef089846400682f59f2, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=e2c09978fa3f44d895cbc17efdca2b32) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:56.853531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5389e2e7-e2e6-4710-900b-0b659da25749 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:56.855075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5389e2e7-e2e6-4710-900b-0b659da25749 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:56.860860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 554/1098] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:56 2022] GET /identity/v3/role_assignments?user.id=e2c09978fa3f44d895cbc17efdca2b32&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=f82ad6f8506b4ef089846400682f59f2 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:57.546354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e83825-273b-4c7d-8004-bfb1db1028cb None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:57.546929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e83825-273b-4c7d-8004-bfb1db1028cb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:57.546929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e83825-273b-4c7d-8004-bfb1db1028cb None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:57.548296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 545/1099] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:58:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:58:57.554015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382d05fd-c7ce-4948-b5da-7beb5a6d432e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:57.554313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382d05fd-c7ce-4948-b5da-7beb5a6d432e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:57.554572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382d05fd-c7ce-4948-b5da-7beb5a6d432e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:57.574112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-382d05fd-c7ce-4948-b5da-7beb5a6d432e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:58:57.575389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-382d05fd-c7ce-4948-b5da-7beb5a6d432e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:58:57.623695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:57.649951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 555/1100] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:58:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:58:57.673516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e83825-273b-4c7d-8004-bfb1db1028cb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:57.780761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e83825-273b-4c7d-8004-bfb1db1028cb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:57.783125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:57.825313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:57.825874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:57.826149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:57.826416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:57.837488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:58:57.842378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:58:57.843865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:58:57.844806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 546/1101] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:58:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:58:57.851033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382d05fd-c7ce-4948-b5da-7beb5a6d432e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:57.966153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382d05fd-c7ce-4948-b5da-7beb5a6d432e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:58.002640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:58.003318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:58.003619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:58.003878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:58.004607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:58.004917 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:58.006309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:58.014840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 556/1102] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:58:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:58.021368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:58.127400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f5ba3f-7fba-41e6-8963-b1d94fe5eee9 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:58.162506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:58.163166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:58.163443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:58.163685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] PATH_INFO: `/v3/users/f051d1f037a9450fae743b092291d152` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:58.164900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] RBAC: Authorizing `identity:get_user(user_id=f051d1f037a9450fae743b092291d152)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:58.166351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:58.168061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 547/1103] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:58:58 2022] GET /identity/v3/users/f051d1f037a9450fae743b092291d152 => generated 285 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:58.174205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:58.216997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:58.284488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad66833-aaea-4d8f-a4c1-195b0909bfcf None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:58.320153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:58.320757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:58.321034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:58.321302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:58.399907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:58.401308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:58.439918 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:58.506355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 557/1104] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:58:58 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:58:58.512800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:58.547163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:58:58.625710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a2236d-7986-4999-8a6b-2bb952290fff None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:58.661781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e045601a-edbc-45d4-93b1-c692585ef138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:58.662344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e045601a-edbc-45d4-93b1-c692585ef138 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:58.662597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e045601a-edbc-45d4-93b1-c692585ef138 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:58.662873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e045601a-edbc-45d4-93b1-c692585ef138 None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:58.664210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e045601a-edbc-45d4-93b1-c692585ef138 None admin] RBAC: Authorizing `identity:get_project(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:58.665653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e045601a-edbc-45d4-93b1-c692585ef138 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:58.667430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 548/1105] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:58:58 2022] GET /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4 => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:58:58.673914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:58:58.775855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09468ee7-b854-43ea-b68d-3824ad66ddb5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:58:58.810720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a162db24-9405-4f67-b95f-e83e00713124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:58:58.811366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162db24-9405-4f67-b95f-e83e00713124 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:58:58.811642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162db24-9405-4f67-b95f-e83e00713124 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:58:58.811913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162db24-9405-4f67-b95f-e83e00713124 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:58:58.812693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a162db24-9405-4f67-b95f-e83e00713124 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4, user.id=f051d1f037a9450fae743b092291d152) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:58:58.813020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a162db24-9405-4f67-b95f-e83e00713124 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:58:58.814461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a162db24-9405-4f67-b95f-e83e00713124 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:58:58.820438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 558/1106] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:58:58 2022] GET /identity/v3/role_assignments?user.id=f051d1f037a9450fae743b092291d152&scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:00.851723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af2d55c-fe1f-44a6-ad4c-623e2e03d88b None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:00.852319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af2d55c-fe1f-44a6-ad4c-623e2e03d88b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:00.852319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af2d55c-fe1f-44a6-ad4c-623e2e03d88b None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:00.853585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 549/1107] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:00.862531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17403080-83f9-4c8c-90c8-8eafdbc6bdd0 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:00.863231 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17403080-83f9-4c8c-90c8-8eafdbc6bdd0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:00.863474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17403080-83f9-4c8c-90c8-8eafdbc6bdd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:00.872643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:00.887512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-17403080-83f9-4c8c-90c8-8eafdbc6bdd0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:00.888710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-17403080-83f9-4c8c-90c8-8eafdbc6bdd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:00.959857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 559/1108] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:00 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:00.985702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af2d55c-fe1f-44a6-ad4c-623e2e03d88b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:01.104400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:01.109467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af2d55c-fe1f-44a6-ad4c-623e2e03d88b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:01.144470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdee7248-868d-4150-8b7b-0217586a3996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:01.145108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdee7248-868d-4150-8b7b-0217586a3996 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:01.145377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdee7248-868d-4150-8b7b-0217586a3996 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:01.145640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdee7248-868d-4150-8b7b-0217586a3996 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:01.146806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdee7248-868d-4150-8b7b-0217586a3996 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:01.148295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdee7248-868d-4150-8b7b-0217586a3996 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:01.149891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 550/1109] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:59:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:01.158833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17403080-83f9-4c8c-90c8-8eafdbc6bdd0 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:01.264998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17403080-83f9-4c8c-90c8-8eafdbc6bdd0 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:01.296702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:01.302630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fb1114-d9b2-4e10-8768-9bb13f3f2271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:01.303280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb1114-d9b2-4e10-8768-9bb13f3f2271 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:01.303558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb1114-d9b2-4e10-8768-9bb13f3f2271 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:01.303789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb1114-d9b2-4e10-8768-9bb13f3f2271 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:01.304496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb1114-d9b2-4e10-8768-9bb13f3f2271 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:01.305929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb1114-d9b2-4e10-8768-9bb13f3f2271 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:01.336138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 560/1110] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:59:01 2022] POST /identity/v3/users => generated 286 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:59:03.403958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c206ba0-2259-48ca-bb88-9299836087bc None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:03.404571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c206ba0-2259-48ca-bb88-9299836087bc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:03.404571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c206ba0-2259-48ca-bb88-9299836087bc None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:03.405875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 551/1111] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:03.412459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab097554-91e9-43d9-b6ce-0d82c5813d6c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:03.413244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab097554-91e9-43d9-b6ce-0d82c5813d6c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:03.413395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab097554-91e9-43d9-b6ce-0d82c5813d6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:03.437889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ab097554-91e9-43d9-b6ce-0d82c5813d6c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:03.439896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ab097554-91e9-43d9-b6ce-0d82c5813d6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:03.525072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 561/1112] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:03.549586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c206ba0-2259-48ca-bb88-9299836087bc None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:03.662987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c206ba0-2259-48ca-bb88-9299836087bc None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:03.700844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:03.701486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:03.701772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:03.702085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] PATH_INFO: `/v3/roles/629444c0e927466f8da9275a0841de95` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:03.703911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] RBAC: Authorizing `identity:get_role(role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:03.705306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:03.706687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 552/1113] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:03 2022] GET /identity/v3/roles/629444c0e927466f8da9275a0841de95 => generated 223 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:03.713241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab097554-91e9-43d9-b6ce-0d82c5813d6c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:03.819223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab097554-91e9-43d9-b6ce-0d82c5813d6c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:03.836898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:03.856179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:03.856815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:03.857086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:03.857355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] PATH_INFO: `/v3/users/b32eeaf6209a435eaca5a24ed83cdaa1` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:03.871841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] RBAC: Authorizing `identity:get_user(user_id=b32eeaf6209a435eaca5a24ed83cdaa1)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:03.873297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:03.874887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 562/1114] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:03 2022] GET /identity/v3/users/b32eeaf6209a435eaca5a24ed83cdaa1 => generated 286 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:03.881358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:03.918041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:03.998620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b90c770-2f00-48db-b09f-e73c21c0dba2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:04.035441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:04.036064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:04.036338 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:04.036604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:04.113784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:04.115270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:04.150099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:04.222709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 553/1115] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:59:03 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:59:04.229503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:04.351585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0104c6d-313f-48b5-96ec-d0275e04d311 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:04.399684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96367e01-f282-4d64-91a2-9d032b85b53d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:04.400333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96367e01-f282-4d64-91a2-9d032b85b53d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:04.400616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96367e01-f282-4d64-91a2-9d032b85b53d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:04.400893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96367e01-f282-4d64-91a2-9d032b85b53d None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:04.402080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96367e01-f282-4d64-91a2-9d032b85b53d None admin] RBAC: Authorizing `identity:get_project(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:04.403575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96367e01-f282-4d64-91a2-9d032b85b53d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:04.405268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 563/1116] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:04 2022] GET /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4 => generated 311 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:04.411790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:04.530499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09a1bcf-f403-498c-b2cf-ca55f4844220 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:04.579715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462963f5-5c69-4605-b4e2-f75b646b070e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:04.580337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462963f5-5c69-4605-b4e2-f75b646b070e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:04.580616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462963f5-5c69-4605-b4e2-f75b646b070e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:04.580889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462963f5-5c69-4605-b4e2-f75b646b070e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:04.581688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462963f5-5c69-4605-b4e2-f75b646b070e None admin] RBAC: Adding query filter params (role.id=629444c0e927466f8da9275a0841de95, scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4, user.id=b32eeaf6209a435eaca5a24ed83cdaa1) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:04.582005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462963f5-5c69-4605-b4e2-f75b646b070e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:04.583510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462963f5-5c69-4605-b4e2-f75b646b070e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:04.586871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:04.590644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 554/1117] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:59:04 2022] GET /identity/v3/role_assignments?user.id=b32eeaf6209a435eaca5a24ed83cdaa1&scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4&role.id=629444c0e927466f8da9275a0841de95 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:06.547778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dd3c15-0593-4364-aff8-8b3fbe8dc560 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:06.548681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dd3c15-0593-4364-aff8-8b3fbe8dc560 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:06.548681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1dd3c15-0593-4364-aff8-8b3fbe8dc560 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:06.550810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 564/1118] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:06.557423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87045d7-bcdb-4832-91e4-6a91ef6e6a40 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:06.558148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87045d7-bcdb-4832-91e4-6a91ef6e6a40 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:06.558273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87045d7-bcdb-4832-91e4-6a91ef6e6a40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:06.588861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a87045d7-bcdb-4832-91e4-6a91ef6e6a40 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:06.590626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a87045d7-bcdb-4832-91e4-6a91ef6e6a40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:06.669004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 555/1119] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:06.691598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dd3c15-0593-4364-aff8-8b3fbe8dc560 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:06.740596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:06.797915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1dd3c15-0593-4364-aff8-8b3fbe8dc560 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:06.848919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:06.849554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:06.849847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:06.850110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] PATH_INFO: `/v3/roles/629444c0e927466f8da9275a0841de95` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:06.851638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] RBAC: Authorizing `identity:get_role(role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:06.853009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:06.854354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 565/1120] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:06 2022] GET /identity/v3/roles/629444c0e927466f8da9275a0841de95 => generated 223 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:06.861276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87045d7-bcdb-4832-91e4-6a91ef6e6a40 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:06.974124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87045d7-bcdb-4832-91e4-6a91ef6e6a40 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:07.003901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:07.010479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:07.011177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:07.011457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:07.011729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] PATH_INFO: `/v3/users/b32eeaf6209a435eaca5a24ed83cdaa1` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:07.013409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] RBAC: Authorizing `identity:get_user(user_id=b32eeaf6209a435eaca5a24ed83cdaa1)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:07.014350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:07.016013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 556/1121] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:06 2022] GET /identity/v3/users/b32eeaf6209a435eaca5a24ed83cdaa1 => generated 286 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:07.022265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:07.126860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fc835b-20d9-46cc-acce-464c8f4f8c6d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:07.140469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:07.163154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ac5237-0bdd-40b3-9f55-74406561df29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:07.163795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac5237-0bdd-40b3-9f55-74406561df29 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:07.164078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac5237-0bdd-40b3-9f55-74406561df29 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:07.164355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac5237-0bdd-40b3-9f55-74406561df29 None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:07.165518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ac5237-0bdd-40b3-9f55-74406561df29 None admin] RBAC: Authorizing `identity:get_project(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:07.167131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ac5237-0bdd-40b3-9f55-74406561df29 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:07.168801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 566/1122] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:07 2022] GET /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:07.178043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:07.297220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef30c104-bddc-4dfe-93da-68171ac166f7 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:07.334388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1d67ad-6eaa-4dfc-b29b-9be428f7dcca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:07.335129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1d67ad-6eaa-4dfc-b29b-9be428f7dcca None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:07.335409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1d67ad-6eaa-4dfc-b29b-9be428f7dcca None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:07.335684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1d67ad-6eaa-4dfc-b29b-9be428f7dcca None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4/users/b32eeaf6209a435eaca5a24ed83cdaa1/roles/629444c0e927466f8da9275a0841de95` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:07.337959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1d67ad-6eaa-4dfc-b29b-9be428f7dcca None admin] RBAC: Authorizing `identity:create_grant(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4, user_id=b32eeaf6209a435eaca5a24ed83cdaa1, role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:07.339496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1d67ad-6eaa-4dfc-b29b-9be428f7dcca None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:07.356072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 557/1123] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:59:07 2022] PUT /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4/users/b32eeaf6209a435eaca5a24ed83cdaa1/roles/629444c0e927466f8da9275a0841de95 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:59:09.327343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684bcb1f-2a4f-42c1-88d8-b9fb6414b1c9 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:09.328177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684bcb1f-2a4f-42c1-88d8-b9fb6414b1c9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:09.328177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684bcb1f-2a4f-42c1-88d8-b9fb6414b1c9 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:09.330245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 567/1124] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:09.339052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20939e1c-999f-4348-a006-0bb2e37638d7 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:09.339454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20939e1c-999f-4348-a006-0bb2e37638d7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:09.339598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20939e1c-999f-4348-a006-0bb2e37638d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:09.359768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-20939e1c-999f-4348-a006-0bb2e37638d7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:09.360969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-20939e1c-999f-4348-a006-0bb2e37638d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:09.431622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:09.452506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 558/1125] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:09 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:09.475840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684bcb1f-2a4f-42c1-88d8-b9fb6414b1c9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:09.590450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684bcb1f-2a4f-42c1-88d8-b9fb6414b1c9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:09.627486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:09.628135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:09.628411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:09.628688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] PATH_INFO: `/v3/roles/629444c0e927466f8da9275a0841de95` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:09.630186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] RBAC: Authorizing `identity:get_role(role_id=629444c0e927466f8da9275a0841de95)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:09.631606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:09.632976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 568/1126] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:09 2022] GET /identity/v3/roles/629444c0e927466f8da9275a0841de95 => generated 223 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:09.639505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20939e1c-999f-4348-a006-0bb2e37638d7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:09.746365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20939e1c-999f-4348-a006-0bb2e37638d7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:09.782721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:09.783391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:09.783666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:09.783942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] PATH_INFO: `/v3/users/b32eeaf6209a435eaca5a24ed83cdaa1` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:09.785224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] RBAC: Authorizing `identity:get_user(user_id=b32eeaf6209a435eaca5a24ed83cdaa1)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:09.786679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:09.788339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 559/1127] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:09 2022] GET /identity/v3/users/b32eeaf6209a435eaca5a24ed83cdaa1 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:09.795288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:09.855476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:09.915818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25c17e6-0295-418a-bc28-820729bb10b5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:09.953707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:09.954329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:09.954611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:09.954886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:10.027864 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:10.029316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:10.077618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:10.133434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 569/1128] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:59:09 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:59:10.143090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:10.234030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:10.289374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7206fed4-25a5-467c-833a-897748182bf0 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:10.330182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd85f67-2894-4fd3-99d2-0bd09e1ee2bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:10.330825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd85f67-2894-4fd3-99d2-0bd09e1ee2bc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:10.331141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd85f67-2894-4fd3-99d2-0bd09e1ee2bc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:10.331413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd85f67-2894-4fd3-99d2-0bd09e1ee2bc None admin] PATH_INFO: `/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:10.332698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd85f67-2894-4fd3-99d2-0bd09e1ee2bc None admin] RBAC: Authorizing `identity:get_project(project_id=d5077bc8ef5546e1bf166ecf2a46e2d4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:10.334143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd85f67-2894-4fd3-99d2-0bd09e1ee2bc None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:10.335859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 560/1129] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:10 2022] GET /identity/v3/projects/d5077bc8ef5546e1bf166ecf2a46e2d4 => generated 311 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:10.346140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:10.453474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338402cd-597e-4aef-afbb-314dc381a210 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:10.491087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9fea6b-adaf-48a7-a064-73efc9cd5d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:10.491694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9fea6b-adaf-48a7-a064-73efc9cd5d8d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:10.491972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9fea6b-adaf-48a7-a064-73efc9cd5d8d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:10.492241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9fea6b-adaf-48a7-a064-73efc9cd5d8d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:10.493037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9fea6b-adaf-48a7-a064-73efc9cd5d8d None admin] RBAC: Adding query filter params (role.id=629444c0e927466f8da9275a0841de95, scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4, user.id=b32eeaf6209a435eaca5a24ed83cdaa1) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:10.493344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9fea6b-adaf-48a7-a064-73efc9cd5d8d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:10.494790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9fea6b-adaf-48a7-a064-73efc9cd5d8d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:10.500734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 570/1130] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:59:10 2022] GET /identity/v3/role_assignments?user.id=b32eeaf6209a435eaca5a24ed83cdaa1&scope.project.id=d5077bc8ef5546e1bf166ecf2a46e2d4&role.id=629444c0e927466f8da9275a0841de95 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:12.515461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6186d-122c-4dba-bf40-f8f80c3fa9ac None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:12.516025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6186d-122c-4dba-bf40-f8f80c3fa9ac None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:12.516025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6186d-122c-4dba-bf40-f8f80c3fa9ac None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:12.518045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 561/1131] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:12.526599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67058b50-1e32-48fe-8be1-5dbf9f3c3319 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:12.527011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67058b50-1e32-48fe-8be1-5dbf9f3c3319 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:12.527236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67058b50-1e32-48fe-8be1-5dbf9f3c3319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:12.552423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-67058b50-1e32-48fe-8be1-5dbf9f3c3319 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:12.553606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-67058b50-1e32-48fe-8be1-5dbf9f3c3319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:12.561937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:12.641380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 571/1132] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:12.672364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d6186d-122c-4dba-bf40-f8f80c3fa9ac None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:12.787101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d6186d-122c-4dba-bf40-f8f80c3fa9ac None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:12.810820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:12.831938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113180b5-b718-437b-be78-295371ac78b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:12.832951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113180b5-b718-437b-be78-295371ac78b6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:12.833418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113180b5-b718-437b-be78-295371ac78b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:12.833868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113180b5-b718-437b-be78-295371ac78b6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:12.836041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113180b5-b718-437b-be78-295371ac78b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:12.838401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113180b5-b718-437b-be78-295371ac78b6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:12.840351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 562/1133] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:59:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:12.847067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67058b50-1e32-48fe-8be1-5dbf9f3c3319 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:12.952791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67058b50-1e32-48fe-8be1-5dbf9f3c3319 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:12.988376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c0aa7b-7eb6-41a6-ab52-f309f7be2db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:12.988988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c0aa7b-7eb6-41a6-ab52-f309f7be2db7 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:12.989266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c0aa7b-7eb6-41a6-ab52-f309f7be2db7 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:12.989539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c0aa7b-7eb6-41a6-ab52-f309f7be2db7 None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:12.990228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c0aa7b-7eb6-41a6-ab52-f309f7be2db7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:12.991701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c0aa7b-7eb6-41a6-ab52-f309f7be2db7 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:13.007820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:13.023309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 572/1134] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:59:12 2022] POST /identity/v3/projects => generated 311 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:59:15.040393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d531eb6-6a99-4097-a92d-9121a505a6a4 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:15.040884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d531eb6-6a99-4097-a92d-9121a505a6a4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:15.040884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d531eb6-6a99-4097-a92d-9121a505a6a4 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:15.042335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 563/1135] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:15.047800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466533ac-c044-49e9-bfe5-51a1c277700c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:15.048249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466533ac-c044-49e9-bfe5-51a1c277700c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:15.048339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466533ac-c044-49e9-bfe5-51a1c277700c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:15.067944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-466533ac-c044-49e9-bfe5-51a1c277700c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:15.069139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-466533ac-c044-49e9-bfe5-51a1c277700c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:15.144607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 573/1136] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:15 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:15.169664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d531eb6-6a99-4097-a92d-9121a505a6a4 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:15.279314 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d531eb6-6a99-4097-a92d-9121a505a6a4 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:15.314668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2cc228-963c-4923-a061-affe5c6888a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:15.315327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2cc228-963c-4923-a061-affe5c6888a9 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:15.315604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2cc228-963c-4923-a061-affe5c6888a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:15.315873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2cc228-963c-4923-a061-affe5c6888a9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:15.317062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2cc228-963c-4923-a061-affe5c6888a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:15.318530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2cc228-963c-4923-a061-affe5c6888a9 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:15.320161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 564/1137] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 02:59:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:15.326571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466533ac-c044-49e9-bfe5-51a1c277700c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:15.429987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466533ac-c044-49e9-bfe5-51a1c277700c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:15.455181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:15.468040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b206b3-04d2-4629-a2b4-a5c63db1e2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:15.468672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b206b3-04d2-4629-a2b4-a5c63db1e2ed None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:15.468949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b206b3-04d2-4629-a2b4-a5c63db1e2ed None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:15.469220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b206b3-04d2-4629-a2b4-a5c63db1e2ed None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:15.469898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b206b3-04d2-4629-a2b4-a5c63db1e2ed None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:15.471360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b206b3-04d2-4629-a2b4-a5c63db1e2ed None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:15.496931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 574/1138] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:59:15 2022] POST /identity/v3/users => generated 286 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:59:17.557633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f0435-abe0-49a1-84cd-094056d616d4 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:17.558220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f0435-abe0-49a1-84cd-094056d616d4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:17.558220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f0435-abe0-49a1-84cd-094056d616d4 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:17.560300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 565/1139] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:17.568064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684b5017-58fc-485a-bc61-92b4b955daf8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:17.568540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684b5017-58fc-485a-bc61-92b4b955daf8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:17.568640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684b5017-58fc-485a-bc61-92b4b955daf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:17.588346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-684b5017-58fc-485a-bc61-92b4b955daf8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:17.589585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-684b5017-58fc-485a-bc61-92b4b955daf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:17.670297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 575/1140] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:17 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:17.705257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0f0435-abe0-49a1-84cd-094056d616d4 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:17.773214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:17.816516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0f0435-abe0-49a1-84cd-094056d616d4 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:17.852166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:17.852795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:17.853072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:17.853346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:17.864360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:59:17.869199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:59:17.870358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:17.871410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:59:17.875128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 566/1141] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:59:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:59:17.877164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684b5017-58fc-485a-bc61-92b4b955daf8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:17.985802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684b5017-58fc-485a-bc61-92b4b955daf8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:17.996716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:18.022611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:18.023254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:18.023539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:18.023812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:18.024534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:18.024841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:18.026278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:18.034312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 576/1142] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:59:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:18.040677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:18.149490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341375f4-6f68-4a91-9a22-2f932d696a3b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:18.170021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:18.185695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:18.186450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:18.186741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:18.187009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] PATH_INFO: `/v3/users/0b5f37959edc4cabb0e9ca35117aaa34` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:18.201028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] RBAC: Authorizing `identity:get_user(user_id=0b5f37959edc4cabb0e9ca35117aaa34)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:18.202476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:18.204188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 567/1143] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:18 2022] GET /identity/v3/users/0b5f37959edc4cabb0e9ca35117aaa34 => generated 286 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:18.211102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:18.338609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26aaffe-6733-419d-a916-ba1640609cda None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:18.383447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:18.384179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:18.384989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:18.385247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:18.429580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:18.457747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:18.459236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:18.561719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 577/1144] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:59:18 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:59:18.568531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:18.689361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a4bcd8-2af5-4609-9259-2a162411f952 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:18.737214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23c7e78-c1b5-4aef-bf11-6f0e01ec367d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:18.737834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23c7e78-c1b5-4aef-bf11-6f0e01ec367d None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:18.738134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23c7e78-c1b5-4aef-bf11-6f0e01ec367d None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:18.738370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23c7e78-c1b5-4aef-bf11-6f0e01ec367d None admin] PATH_INFO: `/v3/projects/d68636bc773946b49745d80a03e57bd3` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:18.739604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23c7e78-c1b5-4aef-bf11-6f0e01ec367d None admin] RBAC: Authorizing `identity:get_project(project_id=d68636bc773946b49745d80a03e57bd3)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:18.741072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23c7e78-c1b5-4aef-bf11-6f0e01ec367d None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:18.742847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 568/1145] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:18 2022] GET /identity/v3/projects/d68636bc773946b49745d80a03e57bd3 => generated 311 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:18.749530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:18.826306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:18.864031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b9a67a-0a64-406b-9847-914bafd3987a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:18.906077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba477c93-ed93-4162-8add-360242b6bc06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:18.906695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba477c93-ed93-4162-8add-360242b6bc06 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:18.906988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba477c93-ed93-4162-8add-360242b6bc06 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:18.907275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba477c93-ed93-4162-8add-360242b6bc06 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:18.908076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba477c93-ed93-4162-8add-360242b6bc06 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=d68636bc773946b49745d80a03e57bd3, user.id=0b5f37959edc4cabb0e9ca35117aaa34) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:18.908391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba477c93-ed93-4162-8add-360242b6bc06 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:18.909834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba477c93-ed93-4162-8add-360242b6bc06 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:18.915668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 578/1146] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:59:18 2022] GET /identity/v3/role_assignments?user.id=0b5f37959edc4cabb0e9ca35117aaa34&scope.project.id=d68636bc773946b49745d80a03e57bd3&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:20.885493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2489f53-d01d-4931-a124-650331b678ef None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:20.886055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2489f53-d01d-4931-a124-650331b678ef None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:20.886055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2489f53-d01d-4931-a124-650331b678ef None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:20.888078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 569/1147] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:20.895406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03640b1-7897-471a-bb7a-3d14a9fb3fec None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:20.895869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03640b1-7897-471a-bb7a-3d14a9fb3fec None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:20.895968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03640b1-7897-471a-bb7a-3d14a9fb3fec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:20.915865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c03640b1-7897-471a-bb7a-3d14a9fb3fec None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:20.917105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c03640b1-7897-471a-bb7a-3d14a9fb3fec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:20.989273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 579/1148] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:20 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:21.012018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2489f53-d01d-4931-a124-650331b678ef None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:21.067476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:21.135390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2489f53-d01d-4931-a124-650331b678ef None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:21.173249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:21.173817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:21.174099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:21.174377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:21.185654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:59:21.190518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:59:21.191569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:59:21.192588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 570/1149] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:59:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:59:21.200800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03640b1-7897-471a-bb7a-3d14a9fb3fec None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:21.283105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:21.329345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03640b1-7897-471a-bb7a-3d14a9fb3fec None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:21.367118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:21.367735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:21.368014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:21.368288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:21.369011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:21.369318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:21.370705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:21.379506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 580/1150] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:59:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:21.387937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:21.515768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f256e4-d44c-41dc-a648-4f8a6a411be7 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:21.523686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:21.556337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:21.556972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:21.557348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:21.557624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] PATH_INFO: `/v3/users/0b5f37959edc4cabb0e9ca35117aaa34` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:21.558827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] RBAC: Authorizing `identity:get_user(user_id=0b5f37959edc4cabb0e9ca35117aaa34)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:21.560314 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:21.561920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 571/1151] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:21 2022] GET /identity/v3/users/0b5f37959edc4cabb0e9ca35117aaa34 => generated 286 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:21.568084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:21.677609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d486406-ce76-474d-97db-92cfd14a2ed8 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:21.703915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:21.713944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a674791-0e66-4556-978b-b6a1db53281c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:21.714579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a674791-0e66-4556-978b-b6a1db53281c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:21.714853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a674791-0e66-4556-978b-b6a1db53281c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:21.715155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a674791-0e66-4556-978b-b6a1db53281c None admin] PATH_INFO: `/v3/projects/d68636bc773946b49745d80a03e57bd3` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:21.716349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a674791-0e66-4556-978b-b6a1db53281c None admin] RBAC: Authorizing `identity:get_project(project_id=d68636bc773946b49745d80a03e57bd3)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:21.718153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a674791-0e66-4556-978b-b6a1db53281c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:21.719839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 581/1152] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:21 2022] GET /identity/v3/projects/d68636bc773946b49745d80a03e57bd3 => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:21.726280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:21.834121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d7150-4c66-40ed-9a68-d3531ced8ec8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:21.878824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9cf66fd-fb22-4b98-a3d2-d45b1b0b2aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:21.879551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cf66fd-fb22-4b98-a3d2-d45b1b0b2aa7 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:21.879829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cf66fd-fb22-4b98-a3d2-d45b1b0b2aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:21.880105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cf66fd-fb22-4b98-a3d2-d45b1b0b2aa7 None admin] PATH_INFO: `/v3/projects/d68636bc773946b49745d80a03e57bd3/users/0b5f37959edc4cabb0e9ca35117aaa34/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:21.882446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cf66fd-fb22-4b98-a3d2-d45b1b0b2aa7 None admin] RBAC: Authorizing `identity:create_grant(project_id=d68636bc773946b49745d80a03e57bd3, user_id=0b5f37959edc4cabb0e9ca35117aaa34, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:21.883952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9cf66fd-fb22-4b98-a3d2-d45b1b0b2aa7 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:21.899845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 572/1153] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:59:21 2022] PUT /identity/v3/projects/d68636bc773946b49745d80a03e57bd3/users/0b5f37959edc4cabb0e9ca35117aaa34/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:59:23.854704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d5a92-1a4b-4f50-9a9c-863b28e498a4 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:23.855608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d5a92-1a4b-4f50-9a9c-863b28e498a4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:23.855849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d5a92-1a4b-4f50-9a9c-863b28e498a4 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:23.857933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 582/1154] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:23.863385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81443478-4514-498c-92c9-3cca24819887 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:23.863804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81443478-4514-498c-92c9-3cca24819887 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:23.863915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81443478-4514-498c-92c9-3cca24819887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:23.883862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-81443478-4514-498c-92c9-3cca24819887 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:23.885037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-81443478-4514-498c-92c9-3cca24819887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:23.912719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:23.956600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 573/1155] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:23.980274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d5a92-1a4b-4f50-9a9c-863b28e498a4 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:24.091149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d5a92-1a4b-4f50-9a9c-863b28e498a4 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:24.127101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:24.127747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:24.128011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:24.128280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:24.140280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:59:24.145179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:59:24.146322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:59:24.147621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 583/1156] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:59:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:59:24.154318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81443478-4514-498c-92c9-3cca24819887 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:24.270117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81443478-4514-498c-92c9-3cca24819887 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:24.306023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:24.306636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:24.306910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:24.307206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:24.307918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:24.308217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:24.309652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:24.317961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 574/1157] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:59:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:24.326685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:24.388538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:24.439131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee37a8c-d5fc-4347-8fa2-c02eafa2b7b9 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:24.477528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:24.478170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:24.478444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:24.478724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] PATH_INFO: `/v3/users/0b5f37959edc4cabb0e9ca35117aaa34` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:24.479968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] RBAC: Authorizing `identity:get_user(user_id=0b5f37959edc4cabb0e9ca35117aaa34)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:24.481463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:24.483095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 584/1158] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:24 2022] GET /identity/v3/users/0b5f37959edc4cabb0e9ca35117aaa34 => generated 286 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:24.489220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:24.517774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:24.593869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dfc478-e00d-43ee-95e4-56b9a210e503 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:24.630510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:24.631176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:24.631431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:24.631737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:24.707107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:24.708487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:24.735305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:24.809988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 575/1159] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:59:24 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:59:24.818780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:24.938370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226481f9-d6cd-498d-8b55-09e1a45616c5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:24.964746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:24.976822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6879a27-0919-4d27-8106-d41c0d670f63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:24.977464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6879a27-0919-4d27-8106-d41c0d670f63 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:24.977743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6879a27-0919-4d27-8106-d41c0d670f63 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:24.978018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6879a27-0919-4d27-8106-d41c0d670f63 None admin] PATH_INFO: `/v3/projects/d68636bc773946b49745d80a03e57bd3` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:24.979270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6879a27-0919-4d27-8106-d41c0d670f63 None admin] RBAC: Authorizing `identity:get_project(project_id=d68636bc773946b49745d80a03e57bd3)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:24.980721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6879a27-0919-4d27-8106-d41c0d670f63 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:24.982427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 585/1160] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:24 2022] GET /identity/v3/projects/d68636bc773946b49745d80a03e57bd3 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:24.988799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:25.089345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ee594c-3bdb-4179-90df-76f91c15e16e None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:25.116407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:25.128443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bcac8a-4fb1-40d4-bec1-799d328c3de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:25.129069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcac8a-4fb1-40d4-bec1-799d328c3de4 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:25.129349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcac8a-4fb1-40d4-bec1-799d328c3de4 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:25.129616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bcac8a-4fb1-40d4-bec1-799d328c3de4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:25.130389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bcac8a-4fb1-40d4-bec1-799d328c3de4 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=d68636bc773946b49745d80a03e57bd3, user.id=0b5f37959edc4cabb0e9ca35117aaa34) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:25.130699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bcac8a-4fb1-40d4-bec1-799d328c3de4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:25.132175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bcac8a-4fb1-40d4-bec1-799d328c3de4 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:25.138552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 576/1161] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:59:24 2022] GET /identity/v3/role_assignments?user.id=0b5f37959edc4cabb0e9ca35117aaa34&scope.project.id=d68636bc773946b49745d80a03e57bd3&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:27.201629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec900b5-6902-4cbc-8566-a6b23d1e3c09 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:27.202886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec900b5-6902-4cbc-8566-a6b23d1e3c09 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:27.203215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec900b5-6902-4cbc-8566-a6b23d1e3c09 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:27.205440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 586/1162] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:27.211062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b9d87c-27ba-4606-ac87-c49bf1002c09 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:27.211515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b9d87c-27ba-4606-ac87-c49bf1002c09 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:27.211606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b9d87c-27ba-4606-ac87-c49bf1002c09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:27.230986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-47b9d87c-27ba-4606-ac87-c49bf1002c09 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:27.232252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-47b9d87c-27ba-4606-ac87-c49bf1002c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:27.305323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 577/1163] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:27 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:27.328533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec900b5-6902-4cbc-8566-a6b23d1e3c09 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:27.443983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec900b5-6902-4cbc-8566-a6b23d1e3c09 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:27.479840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:27.480447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:27.480779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:27.481051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:27.544746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:27.556491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:27.557879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:27.664258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 587/1164] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:59:27 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:59:27.672243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b9d87c-27ba-4606-ac87-c49bf1002c09 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:27.787563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b9d87c-27ba-4606-ac87-c49bf1002c09 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:27.809872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:27.824270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bd5cb2-f96d-4c3a-bd11-41250afeed00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:27.824940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd5cb2-f96d-4c3a-bd11-41250afeed00 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:27.825220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd5cb2-f96d-4c3a-bd11-41250afeed00 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:27.825490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd5cb2-f96d-4c3a-bd11-41250afeed00 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:27.832069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bd5cb2-f96d-4c3a-bd11-41250afeed00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:27.833516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bd5cb2-f96d-4c3a-bd11-41250afeed00 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-b8bd5cb2-f96d-4c3a-bd11-41250afeed00 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:59:27.845120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:59:27.846316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:59:27.847717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 578/1165] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 02:59:27 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:59:27.852953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:27.950034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:27.960807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450339b8-4c4a-495d-a86c-a1f40a5ef5b3 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:27.997667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e2992c-eb9b-4ac8-8dff-bcb935e516de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:27.998293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2992c-eb9b-4ac8-8dff-bcb935e516de None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:27.998569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2992c-eb9b-4ac8-8dff-bcb935e516de None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:27.998818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2992c-eb9b-4ac8-8dff-bcb935e516de None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:27.999566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2992c-eb9b-4ac8-8dff-bcb935e516de None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:27.999867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2992c-eb9b-4ac8-8dff-bcb935e516de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:28.001243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2992c-eb9b-4ac8-8dff-bcb935e516de None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:28.009315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 588/1166] 10.222.0.42 () {56 vars in 1115 bytes} [Tue Jul 26 02:59:27 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:29.986170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf4e236-2103-4a15-95a9-0b776487affa None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:29.986773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf4e236-2103-4a15-95a9-0b776487affa None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:29.986773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf4e236-2103-4a15-95a9-0b776487affa None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:29.988116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 579/1167] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:29.994538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fbb4b-b73e-42de-bd63-f085817639c5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:29.995034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fbb4b-b73e-42de-bd63-f085817639c5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:29.995148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fbb4b-b73e-42de-bd63-f085817639c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:30.020215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ef6fbb4b-b73e-42de-bd63-f085817639c5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:30.022024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ef6fbb4b-b73e-42de-bd63-f085817639c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:30.094914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 589/1168] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:29 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:30.118429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf4e236-2103-4a15-95a9-0b776487affa None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:30.233061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf4e236-2103-4a15-95a9-0b776487affa None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:30.267681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338858a5-7e37-4eaf-ad42-59ad97e8a654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:30.268223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338858a5-7e37-4eaf-ad42-59ad97e8a654 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:30.268496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338858a5-7e37-4eaf-ad42-59ad97e8a654 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:30.268763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338858a5-7e37-4eaf-ad42-59ad97e8a654 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:30.269446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338858a5-7e37-4eaf-ad42-59ad97e8a654 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:30.270827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338858a5-7e37-4eaf-ad42-59ad97e8a654 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:30.290330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 580/1169] 10.222.0.42 () {60 vars in 1138 bytes} [Tue Jul 26 02:59:30 2022] POST /identity/v3/domains => generated 267 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:59:32.315191 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413d64c2-0e66-4355-a230-06ed18fbba0a None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:32.315726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413d64c2-0e66-4355-a230-06ed18fbba0a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:32.315726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413d64c2-0e66-4355-a230-06ed18fbba0a None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:32.317619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 590/1170] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:32.326167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0f97a1-3830-4196-b3b3-95b0e9d91de4 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:32.326913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0f97a1-3830-4196-b3b3-95b0e9d91de4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:32.327054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0f97a1-3830-4196-b3b3-95b0e9d91de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:32.351282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-dd0f97a1-3830-4196-b3b3-95b0e9d91de4 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:32.352488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dd0f97a1-3830-4196-b3b3-95b0e9d91de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:32.390248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:32.423042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 581/1171] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:32.450611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413d64c2-0e66-4355-a230-06ed18fbba0a None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:32.565331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:32.584161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413d64c2-0e66-4355-a230-06ed18fbba0a None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:32.622197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff564ff6-c2d8-461a-89eb-db4ff51b2490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:32.622849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff564ff6-c2d8-461a-89eb-db4ff51b2490 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:32.623164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff564ff6-c2d8-461a-89eb-db4ff51b2490 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:32.623436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff564ff6-c2d8-461a-89eb-db4ff51b2490 None admin] PATH_INFO: `/v3/domains/e3edb271208b4375b56a68551c6461c7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:32.636093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff564ff6-c2d8-461a-89eb-db4ff51b2490 None admin] RBAC: Authorizing `identity:get_domain(domain_id=e3edb271208b4375b56a68551c6461c7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:32.637610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff564ff6-c2d8-461a-89eb-db4ff51b2490 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:32.639431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 591/1172] 10.222.0.42 () {56 vars in 1183 bytes} [Tue Jul 26 02:59:32 2022] GET /identity/v3/domains/e3edb271208b4375b56a68551c6461c7 => generated 267 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:32.646181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0f97a1-3830-4196-b3b3-95b0e9d91de4 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:32.760544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0f97a1-3830-4196-b3b3-95b0e9d91de4 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:32.796429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec58b1a-0dab-4e31-8bca-a42af02170d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:32.797049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec58b1a-0dab-4e31-8bca-a42af02170d6 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:32.797331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec58b1a-0dab-4e31-8bca-a42af02170d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:32.797594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec58b1a-0dab-4e31-8bca-a42af02170d6 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:32.798291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec58b1a-0dab-4e31-8bca-a42af02170d6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:32.799771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec58b1a-0dab-4e31-8bca-a42af02170d6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:32.829949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 582/1173] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 02:59:32 2022] POST /identity/v3/projects => generated 361 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 02:59:34.897795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404aec5c-760d-48c6-bad3-a35d820a35b2 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:34.898541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404aec5c-760d-48c6-bad3-a35d820a35b2 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:34.898541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404aec5c-760d-48c6-bad3-a35d820a35b2 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:34.900223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 592/1174] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:34.906427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a75dbcd-d60c-4266-9dc0-9754fecea320 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:34.906924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a75dbcd-d60c-4266-9dc0-9754fecea320 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:34.907064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a75dbcd-d60c-4266-9dc0-9754fecea320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:34.931187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4a75dbcd-d60c-4266-9dc0-9754fecea320 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:34.932380 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4a75dbcd-d60c-4266-9dc0-9754fecea320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:34.951649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:35.017614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 583/1175] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:34 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:35.042310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404aec5c-760d-48c6-bad3-a35d820a35b2 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:35.160355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404aec5c-760d-48c6-bad3-a35d820a35b2 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:35.210001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbf33cd-bd5b-4717-9708-d3ab6598e930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:35.210648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf33cd-bd5b-4717-9708-d3ab6598e930 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:35.210926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf33cd-bd5b-4717-9708-d3ab6598e930 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:35.211230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf33cd-bd5b-4717-9708-d3ab6598e930 None admin] PATH_INFO: `/v3/domains/e3edb271208b4375b56a68551c6461c7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:35.212522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbf33cd-bd5b-4717-9708-d3ab6598e930 None admin] RBAC: Authorizing `identity:get_domain(domain_id=e3edb271208b4375b56a68551c6461c7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:35.213970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbf33cd-bd5b-4717-9708-d3ab6598e930 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:35.215740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 593/1176] 10.222.0.42 () {56 vars in 1183 bytes} [Tue Jul 26 02:59:35 2022] GET /identity/v3/domains/e3edb271208b4375b56a68551c6461c7 => generated 267 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:35.222217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a75dbcd-d60c-4266-9dc0-9754fecea320 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:35.322941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a75dbcd-d60c-4266-9dc0-9754fecea320 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:35.358619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deb765a-86c0-48b7-92d4-1a290e53cc33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:35.359265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deb765a-86c0-48b7-92d4-1a290e53cc33 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:35.359541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deb765a-86c0-48b7-92d4-1a290e53cc33 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:35.359815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deb765a-86c0-48b7-92d4-1a290e53cc33 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:35.360546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5deb765a-86c0-48b7-92d4-1a290e53cc33 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:35.361990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5deb765a-86c0-48b7-92d4-1a290e53cc33 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:35.386491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 584/1177] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 02:59:35 2022] POST /identity/v3/users => generated 311 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 02:59:37.418555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d30b02-df1a-4e19-a42f-d22f61e2b63b None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:37.419460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d30b02-df1a-4e19-a42f-d22f61e2b63b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:37.419460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d30b02-df1a-4e19-a42f-d22f61e2b63b None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:37.421690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 594/1178] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:37.428001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e01a35-3bec-412b-8384-436c1f4e0e8b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:37.428768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e01a35-3bec-412b-8384-436c1f4e0e8b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:37.428888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e01a35-3bec-412b-8384-436c1f4e0e8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:37.450964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-17e01a35-3bec-412b-8384-436c1f4e0e8b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:37.452179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-17e01a35-3bec-412b-8384-436c1f4e0e8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:37.454176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:37.527467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 585/1179] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:37 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:37.564309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d30b02-df1a-4e19-a42f-d22f61e2b63b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:37.588718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:37.685342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d30b02-df1a-4e19-a42f-d22f61e2b63b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:37.720852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:37.721486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:37.721762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:37.722078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:37.733396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:59:37.738258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:59:37.739402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:59:37.740308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 595/1180] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:59:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:59:37.745820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e01a35-3bec-412b-8384-436c1f4e0e8b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:37.846040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e01a35-3bec-412b-8384-436c1f4e0e8b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:37.879494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:37.880118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:37.880395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:37.880664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:37.881384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:37.881687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:37.883105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:37.888143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:37.892457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 586/1181] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:59:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:37.897927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:37.975088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:38.000030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e438e14-6380-43ed-bc6e-85053a2eaae5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:38.046108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:38.046763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:38.047098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:38.047380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] PATH_INFO: `/v3/users/9db67c6727734d5e990a18741c276c1d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:38.060673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] RBAC: Authorizing `identity:get_user(user_id=9db67c6727734d5e990a18741c276c1d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:38.062120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:38.063788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 596/1182] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:37 2022] GET /identity/v3/users/9db67c6727734d5e990a18741c276c1d => generated 311 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:38.069989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:38.172626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb025d7-2f35-4efa-908c-987e3a47aace None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:38.207653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:38.208277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:38.208550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:38.208815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:38.275511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:38.278441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:38.280082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:38.394433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 587/1183] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:59:38 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:59:38.400644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:38.517432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a603071-c88f-4d1f-8edb-fcf13b77f5c4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:38.556321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:38.562634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a970c3-bb7a-433d-8cc1-15b7825bfd12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:38.563317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a970c3-bb7a-433d-8cc1-15b7825bfd12 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:38.563597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a970c3-bb7a-433d-8cc1-15b7825bfd12 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:38.563869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a970c3-bb7a-433d-8cc1-15b7825bfd12 None admin] PATH_INFO: `/v3/projects/1bac5a96a96f44058eaea7facd45abf1` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:38.565020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a970c3-bb7a-433d-8cc1-15b7825bfd12 None admin] RBAC: Authorizing `identity:get_project(project_id=1bac5a96a96f44058eaea7facd45abf1)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:38.566479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a970c3-bb7a-433d-8cc1-15b7825bfd12 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:38.568209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 597/1184] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:38 2022] GET /identity/v3/projects/1bac5a96a96f44058eaea7facd45abf1 => generated 361 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:38.574982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:38.679414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:38.682841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88de21fa-ab24-414a-b536-16e98e2dc560 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:38.719548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0175002b-e8c4-400a-ad26-5743b074f7bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:38.720159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0175002b-e8c4-400a-ad26-5743b074f7bc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:38.720434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0175002b-e8c4-400a-ad26-5743b074f7bc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:38.720704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0175002b-e8c4-400a-ad26-5743b074f7bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:38.721485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0175002b-e8c4-400a-ad26-5743b074f7bc None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=1bac5a96a96f44058eaea7facd45abf1, user.id=9db67c6727734d5e990a18741c276c1d) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:38.721795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0175002b-e8c4-400a-ad26-5743b074f7bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:38.723262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0175002b-e8c4-400a-ad26-5743b074f7bc None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:38.729027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 588/1185] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:59:38 2022] GET /identity/v3/role_assignments?user.id=9db67c6727734d5e990a18741c276c1d&scope.project.id=1bac5a96a96f44058eaea7facd45abf1&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:40.692736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0f849-99be-4d4f-a6f6-8303c07092ce None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:40.693295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0f849-99be-4d4f-a6f6-8303c07092ce None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:40.693295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a0f849-99be-4d4f-a6f6-8303c07092ce None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:40.694641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 598/1186] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 02:59:40.701069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2111f9-00df-4bc8-a860-15aec60dd5cd None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:40.701763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2111f9-00df-4bc8-a860-15aec60dd5cd None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:40.701923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2111f9-00df-4bc8-a860-15aec60dd5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:40.727051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2d2111f9-00df-4bc8-a860-15aec60dd5cd None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:40.728242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2d2111f9-00df-4bc8-a860-15aec60dd5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:40.800671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 589/1187] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:40.826837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a0f849-99be-4d4f-a6f6-8303c07092ce None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:40.947414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a0f849-99be-4d4f-a6f6-8303c07092ce None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:40.989933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:40.990574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:40.990853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:40.991167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:41.006306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:59:41.011976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:59:41.012969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 02:59:41.013820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 599/1188] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:59:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:59:41.019844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2111f9-00df-4bc8-a860-15aec60dd5cd None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:41.127869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:41.136892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2111f9-00df-4bc8-a860-15aec60dd5cd None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:41.171534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:41.172162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:41.172437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:41.172713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:41.173436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:41.173744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:41.175176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:41.183433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 590/1189] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:59:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:41.189662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:41.253362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:41.296788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d46b9ab-0033-4ec3-9457-042f8fccf927 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:41.331794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:41.332424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:41.332701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:41.332971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] PATH_INFO: `/v3/users/9db67c6727734d5e990a18741c276c1d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:41.334249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] RBAC: Authorizing `identity:get_user(user_id=9db67c6727734d5e990a18741c276c1d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:41.335713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:41.337302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 600/1190] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:41 2022] GET /identity/v3/users/9db67c6727734d5e990a18741c276c1d => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:41.343994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:41.445945 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49064e9f-5454-4e60-ad66-c3e410f79721 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:41.481128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f0600b-e890-45fa-96fc-4f1bf59211fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:41.481768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0600b-e890-45fa-96fc-4f1bf59211fb None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:41.482040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0600b-e890-45fa-96fc-4f1bf59211fb None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:41.482311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f0600b-e890-45fa-96fc-4f1bf59211fb None admin] PATH_INFO: `/v3/projects/1bac5a96a96f44058eaea7facd45abf1` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:41.483590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f0600b-e890-45fa-96fc-4f1bf59211fb None admin] RBAC: Authorizing `identity:get_project(project_id=1bac5a96a96f44058eaea7facd45abf1)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:41.485030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f0600b-e890-45fa-96fc-4f1bf59211fb None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:41.486692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 591/1191] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:41 2022] GET /identity/v3/projects/1bac5a96a96f44058eaea7facd45abf1 => generated 361 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 02:59:41.493706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:41.601138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02511c4c-5eb1-455d-83fe-4920ffce7803 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:41.631186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:41.640628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63804f50-756c-4f7f-b89c-e2f13c30f3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:41.641302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63804f50-756c-4f7f-b89c-e2f13c30f3b8 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:41.641580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63804f50-756c-4f7f-b89c-e2f13c30f3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:41.641856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63804f50-756c-4f7f-b89c-e2f13c30f3b8 None admin] PATH_INFO: `/v3/projects/1bac5a96a96f44058eaea7facd45abf1/users/9db67c6727734d5e990a18741c276c1d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:41.643861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63804f50-756c-4f7f-b89c-e2f13c30f3b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=1bac5a96a96f44058eaea7facd45abf1, user_id=9db67c6727734d5e990a18741c276c1d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:41.645321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63804f50-756c-4f7f-b89c-e2f13c30f3b8 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:41.660807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 601/1192] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 02:59:41 2022] PUT /identity/v3/projects/1bac5a96a96f44058eaea7facd45abf1/users/9db67c6727734d5e990a18741c276c1d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 02:59:43.635224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f4653f-341d-4e93-a61a-7100330a7d79 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:43.635786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f4653f-341d-4e93-a61a-7100330a7d79 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:43.635786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f4653f-341d-4e93-a61a-7100330a7d79 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:43.637189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 592/1193] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 02:59:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 02:59:43.643583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4d7c2-83ae-467e-b9b2-78d9faf8606d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:43.644280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4d7c2-83ae-467e-b9b2-78d9faf8606d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:43.644461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4d7c2-83ae-467e-b9b2-78d9faf8606d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:43.667309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-70a4d7c2-83ae-467e-b9b2-78d9faf8606d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 02:59:43.668505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-70a4d7c2-83ae-467e-b9b2-78d9faf8606d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 02:59:43.741105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 602/1194] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 02:59:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 02:59:43.765446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f4653f-341d-4e93-a61a-7100330a7d79 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:43.793556 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:43.877782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f4653f-341d-4e93-a61a-7100330a7d79 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:43.915527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:43.916099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:43.916376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:43.916650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:43.927870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 02:59:43.932714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 02:59:43.933803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 02:59:43.934716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 593/1195] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 02:59:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 02:59:43.940348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a4d7c2-83ae-467e-b9b2-78d9faf8606d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:44.044164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a4d7c2-83ae-467e-b9b2-78d9faf8606d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:44.061919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:44.082542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:44.083175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:44.083442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:44.083716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:44.084463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:44.084739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:44.086137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:44.095341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 603/1196] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 02:59:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:44.101715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:44.191935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:44.208399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20ba71a-f1ee-46bf-893f-3cedf59bde58 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:44.243908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:44.244558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:44.244827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:44.245099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] PATH_INFO: `/v3/users/9db67c6727734d5e990a18741c276c1d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:44.246361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] RBAC: Authorizing `identity:get_user(user_id=9db67c6727734d5e990a18741c276c1d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:44.247848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:44.249463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 594/1197] 10.222.0.42 () {56 vars in 1177 bytes} [Tue Jul 26 02:59:44 2022] GET /identity/v3/users/9db67c6727734d5e990a18741c276c1d => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:44.255700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:44.361341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d98b529-0502-41ea-be0f-5d6826959852 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:44.397461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:44.398078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:44.398356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:44.398624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:44.449902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:44.473328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:44.474809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:44.586171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 604/1198] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 02:59:44 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 02:59:44.592826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:44.704887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49559837-6341-4e9c-a136-c83db337db52 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:44.740199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ad9c6d-7ec9-4aeb-a1d0-f120260ea8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:44.740835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ad9c6d-7ec9-4aeb-a1d0-f120260ea8cd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:44.741109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ad9c6d-7ec9-4aeb-a1d0-f120260ea8cd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:44.741376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ad9c6d-7ec9-4aeb-a1d0-f120260ea8cd None admin] PATH_INFO: `/v3/projects/1bac5a96a96f44058eaea7facd45abf1` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:44.742541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ad9c6d-7ec9-4aeb-a1d0-f120260ea8cd None admin] RBAC: Authorizing `identity:get_project(project_id=1bac5a96a96f44058eaea7facd45abf1)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:44.744011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ad9c6d-7ec9-4aeb-a1d0-f120260ea8cd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:44.745806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 595/1199] 10.222.0.42 () {56 vars in 1186 bytes} [Tue Jul 26 02:59:44 2022] GET /identity/v3/projects/1bac5a96a96f44058eaea7facd45abf1 => generated 361 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:44.752378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 02:59:44.843400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:44.864611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dfc173-39fe-42af-8253-d92db87b2305 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 02:59:44.903383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8a1426-b9ad-4f6e-8bde-ccabd7c8325e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 02:59:44.903992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8a1426-b9ad-4f6e-8bde-ccabd7c8325e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 02:59:44.904272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8a1426-b9ad-4f6e-8bde-ccabd7c8325e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 02:59:44.904543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8a1426-b9ad-4f6e-8bde-ccabd7c8325e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 02:59:44.905343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8a1426-b9ad-4f6e-8bde-ccabd7c8325e None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=1bac5a96a96f44058eaea7facd45abf1, user.id=9db67c6727734d5e990a18741c276c1d) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 02:59:44.905651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8a1426-b9ad-4f6e-8bde-ccabd7c8325e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 02:59:44.907117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8a1426-b9ad-4f6e-8bde-ccabd7c8325e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 02:59:44.913357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 605/1200] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 02:59:44 2022] GET /identity/v3/role_assignments?user.id=9db67c6727734d5e990a18741c276c1d&scope.project.id=1bac5a96a96f44058eaea7facd45abf1&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 02:59:54.735438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 02:59:54.910256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=28 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:40.783625 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120ab58-67c2-48f7-9d8e-e3ca7b5bcb6d None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:40.784273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120ab58-67c2-48f7-9d8e-e3ca7b5bcb6d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:40.784273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120ab58-67c2-48f7-9d8e-e3ca7b5bcb6d None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:40.785734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 596/1201] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:00:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:00:40.794267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fff192-6c47-419c-9c77-e1d9bcf33882 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:40.794267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fff192-6c47-419c-9c77-e1d9bcf33882 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:40.794267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fff192-6c47-419c-9c77-e1d9bcf33882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:40.813771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-88fff192-6c47-419c-9c77-e1d9bcf33882 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:00:40.815438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-88fff192-6c47-419c-9c77-e1d9bcf33882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:00:40.898536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 606/1202] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:00:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:00:40.925761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9120ab58-67c2-48f7-9d8e-e3ca7b5bcb6d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:41.066671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9120ab58-67c2-48f7-9d8e-e3ca7b5bcb6d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:41.110483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:41.110483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:41.110801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:41.110888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:41.125498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:41.127099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:41.145737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:41.147219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:41.148198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:41.148198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:41.148198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:41.148198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:41.148198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 597/1203] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:41.153428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fff192-6c47-419c-9c77-e1d9bcf33882 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:41.313438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fff192-6c47-419c-9c77-e1d9bcf33882 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:41.351781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:41.359594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e5f56c-1a92-47a4-9b29-75cba26dc9cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:41.360513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e5f56c-1a92-47a4-9b29-75cba26dc9cc None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:41.360940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e5f56c-1a92-47a4-9b29-75cba26dc9cc None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:41.361402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e5f56c-1a92-47a4-9b29-75cba26dc9cc None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:41.362121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e5f56c-1a92-47a4-9b29-75cba26dc9cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:41.362421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e5f56c-1a92-47a4-9b29-75cba26dc9cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:41.363920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e5f56c-1a92-47a4-9b29-75cba26dc9cc None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:41.377215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 607/1204] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:41.384359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:41.456785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:41.511797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8069def-130a-4f45-854d-c11b744ca07a None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:41.556612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aabf672-ce48-4e28-9fd1-513a617674a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:41.557237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aabf672-ce48-4e28-9fd1-513a617674a1 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:41.557440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aabf672-ce48-4e28-9fd1-513a617674a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:41.557718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aabf672-ce48-4e28-9fd1-513a617674a1 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:41.558455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aabf672-ce48-4e28-9fd1-513a617674a1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:41.559951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aabf672-ce48-4e28-9fd1-513a617674a1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:41.592478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 598/1205] 10.222.0.42 () {60 vars in 1132 bytes} [Tue Jul 26 03:00:41 2022] POST /identity/v3/users => generated 251 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:00:43.878011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f78e2b-6207-421a-9791-13ca34c68a32 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:43.878687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f78e2b-6207-421a-9791-13ca34c68a32 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:43.878687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f78e2b-6207-421a-9791-13ca34c68a32 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:43.882690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 608/1206] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:00:43 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:00:43.889866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeddcc6-ada6-4a1d-841a-2411b5f68981 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:43.890386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeddcc6-ada6-4a1d-841a-2411b5f68981 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:43.890479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeddcc6-ada6-4a1d-841a-2411b5f68981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:43.915942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2eeddcc6-ada6-4a1d-841a-2411b5f68981 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:00:43.916704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2eeddcc6-ada6-4a1d-841a-2411b5f68981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:00:44.023611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 599/1207] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:00:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 03:00:44.048384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f78e2b-6207-421a-9791-13ca34c68a32 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:44.197647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f78e2b-6207-421a-9791-13ca34c68a32 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:44.241386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:44.242112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:44.242414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:44.242665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:44.258730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 03:00:44.264184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:44.265502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:44.266398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 609/1208] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 03:00:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:44.273127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeddcc6-ada6-4a1d-841a-2411b5f68981 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:44.325629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:44.399268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eeddcc6-ada6-4a1d-841a-2411b5f68981 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:44.438245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:44.438936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:44.439241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:44.439633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:44.440420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:44.440730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:44.442292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:44.458672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 600/1209] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 03:00:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:44.465142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:44.521810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:44.576808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e98be-b93e-418b-9868-7386846162fb None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:44.619539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:44.620611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:44.621122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:44.621483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:44.641373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:44.642919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:44.665555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:44.666566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:44.667528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:44.667528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:44.667528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:44.667528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:44.667528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 610/1210] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:44.677309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:44.858137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:44.862192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d95f7b-38df-47e7-8bea-005606f9712b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:44.909320 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:44.910501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:44.910501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:44.910501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:44.911240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:44.911616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:44.913092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:44.926313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 601/1211] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:44.933849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:45.071008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31dbdd-efb9-438c-b557-0d2fe2fa2ca1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:45.073019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:45.127887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:45.128876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:45.129314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:45.129756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:45.145883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:45.148974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 03:00:45.164336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:45.165396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:45.166355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 611/1212] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 03:00:45.174296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:45.307873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f8c5c8-14c7-4015-ad6b-efa4c8bd0382 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:45.358829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:45.359512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:45.359796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:45.360204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:45.361087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:45.362055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:45.363053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:45.389566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 602/1213] 10.222.0.42 () {56 vars in 1143 bytes} [Tue Jul 26 03:00:45 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:00:45.390713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:45.511521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2110e14-e4d7-4794-879a-368906894df9 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:45.550932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:45.551546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:45.551839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:45.552116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:45.589769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:45.642929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:45.645371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:45.777916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 612/1214] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 03:00:45 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 03:00:45.784291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:45.815153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:45.910484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a0ca06-264e-4131-b01e-1cf2b6b40d68 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:45.955370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:45.956163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:45.956450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:45.956718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:45.970159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:45.972011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:45.989461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:45.990775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:45.991849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:45.991849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:45.991849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:45.991849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:45.991849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 603/1215] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 03:00:45.997479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:46.079331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:46.131815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0758839b-cd31-4fe0-84f6-d9f390cecf9c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:46.169596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:46.170205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:46.170511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:46.170818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:46.171564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:46.171894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:46.173335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:46.191788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 613/1216] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:46.193218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:46.278406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:46.313445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756cf5d-a850-47b8-a4f0-9a6752f0a79f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:46.354883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:46.355711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:46.356084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:46.356451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:46.363720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:46.365286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 03:00:46.377132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:46.379679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:46.380563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 604/1217] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 03:00:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 03:00:46.384713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:46.504072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a77c23-90a5-4cbe-b86c-ac5ba3ed7f32 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:46.528114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:46.539919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b389c2b-5ed6-4812-85a2-27c9f088d75c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:46.540619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b389c2b-5ed6-4812-85a2-27c9f088d75c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:46.540896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b389c2b-5ed6-4812-85a2-27c9f088d75c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:46.541168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b389c2b-5ed6-4812-85a2-27c9f088d75c None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:46.541930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b389c2b-5ed6-4812-85a2-27c9f088d75c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:46.542235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b389c2b-5ed6-4812-85a2-27c9f088d75c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:46.543700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b389c2b-5ed6-4812-85a2-27c9f088d75c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:46.556772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 614/1218] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 03:00:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:46.563729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:46.676979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34686d6-3552-4e20-9a39-fa2f31aeca34 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:46.708166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:46.717377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078c7b16-182a-4ca1-aaef-dc3e965e6ee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:46.718214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c7b16-182a-4ca1-aaef-dc3e965e6ee1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:46.718518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c7b16-182a-4ca1-aaef-dc3e965e6ee1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:46.718956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c7b16-182a-4ca1-aaef-dc3e965e6ee1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:46.719757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078c7b16-182a-4ca1-aaef-dc3e965e6ee1 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=35f28f9458e54fb8a76c740d6f9c33bf) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:46.720206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078c7b16-182a-4ca1-aaef-dc3e965e6ee1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:46.721780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078c7b16-182a-4ca1-aaef-dc3e965e6ee1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:46.735070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 605/1219] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 03:00:46 2022] GET /identity/v3/role_assignments?user.id=35f28f9458e54fb8a76c740d6f9c33bf&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:48.910139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bcf23d-a42d-4194-a814-24568deedb85 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:48.910748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bcf23d-a42d-4194-a814-24568deedb85 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:48.910748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bcf23d-a42d-4194-a814-24568deedb85 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:48.912154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 615/1220] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:00:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:00:48.918182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdc94d2-0ab4-428e-90dd-da2cd1364685 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:48.918660 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdc94d2-0ab4-428e-90dd-da2cd1364685 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:48.918740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdc94d2-0ab4-428e-90dd-da2cd1364685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:48.938368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ebdc94d2-0ab4-428e-90dd-da2cd1364685 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:00:48.939735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ebdc94d2-0ab4-428e-90dd-da2cd1364685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:00:49.019120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 606/1221] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:00:48 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:00:49.045015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bcf23d-a42d-4194-a814-24568deedb85 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:49.163498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bcf23d-a42d-4194-a814-24568deedb85 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:49.201593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:49.203581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:49.203887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:49.204169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:49.216675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 03:00:49.221714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:49.222879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:49.223778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 616/1222] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 03:00:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:49.229135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdc94d2-0ab4-428e-90dd-da2cd1364685 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:49.339272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdc94d2-0ab4-428e-90dd-da2cd1364685 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:49.364842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:49.376904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:49.377541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:49.377820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:49.378091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:49.378864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] RBAC: Adding query filter params (name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:49.379148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:49.380537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:49.388594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 607/1223] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 03:00:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:49.394531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:49.445176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:49.506509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f32d750-09b6-425a-9622-c10e3a385988 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:49.546053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:49.546804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:49.547125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:49.547410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:49.558752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:49.560436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:49.575969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:49.577201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:49.578333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:49.578333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:49.578333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:49.578333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:49.578333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 617/1224] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:49.583258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:49.701049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eef5732-3d3d-4665-a9fa-473a882d21c3 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:49.746583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:49.747295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:49.747577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:49.747846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:49.748563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:49.748842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:49.750224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:49.762548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 608/1225] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:49.768593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:49.885937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:49.894041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659efaf1-4650-461c-9b19-ab09be225431 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:49.932964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:49.933640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:49.933875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:49.934144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:49.942154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:49.943652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 03:00:49.956167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:49.957337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:49.958445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 618/1226] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:49 2022] GET /identity/v3/users/placement => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:49.963425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:50.014895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:50.069329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1692bb1-c837-47de-b781-5fa87335121b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:50.105627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:50.106275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:50.106555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:50.106857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:50.107720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:50.108053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:50.109591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:50.127551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 609/1227] 10.222.0.42 () {56 vars in 1143 bytes} [Tue Jul 26 03:00:49 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:50.133881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:50.240251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cfeb85-abd3-4b39-937b-3f6282bd8574 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:50.278056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:50.278767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:50.279104 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:50.279376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:50.290699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:50.292176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:50.307705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:50.308783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:50.315088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:50.315088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:50.315088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:50.315088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:50.315088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 619/1228] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:50.315555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:50.428758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a60ceca-1928-4ef1-96f3-8988b0d345b3 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:50.431387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:50.468443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:50.469123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:50.469425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:50.469828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:50.470606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:50.470939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:50.472523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:50.490580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 610/1229] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:00:50.496587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:50.507669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:50.606681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19054aac-1aa3-44dd-a750-7bd5f60a7c44 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:50.644784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:50.645474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:50.645759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:50.646034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:50.653566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:50.655190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 03:00:50.668098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:50.669251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:50.670251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 620/1230] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 03:00:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:50.676284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:50.803642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dff1b3-9f9b-4207-9c97-4f1f3acca0f5 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:50.841849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be133b6f-6528-477a-af77-33ec234cf5a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:50.842533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be133b6f-6528-477a-af77-33ec234cf5a8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:50.842813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be133b6f-6528-477a-af77-33ec234cf5a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:50.843175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be133b6f-6528-477a-af77-33ec234cf5a8 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:50.843867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be133b6f-6528-477a-af77-33ec234cf5a8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:50.844222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be133b6f-6528-477a-af77-33ec234cf5a8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:50.845614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be133b6f-6528-477a-af77-33ec234cf5a8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:50.858450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 611/1231] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 03:00:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:50.865290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:50.952311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:50.993025 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b70fd4c-edf5-47d5-a3fd-17307319b9b9 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:51.048070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc589ac6-f051-4ab4-b3cd-20645a4997a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:51.049020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc589ac6-f051-4ab4-b3cd-20645a4997a9 None admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:51.049377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc589ac6-f051-4ab4-b3cd-20645a4997a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:51.049690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc589ac6-f051-4ab4-b3cd-20645a4997a9 None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/35f28f9458e54fb8a76c740d6f9c33bf/roles/262cff71501a4fd3b5d8663bc982bde7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:51.072951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc589ac6-f051-4ab4-b3cd-20645a4997a9 None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=35f28f9458e54fb8a76c740d6f9c33bf, role_id=262cff71501a4fd3b5d8663bc982bde7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:51.075001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc589ac6-f051-4ab4-b3cd-20645a4997a9 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:51.094120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 621/1232] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 03:00:50 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/35f28f9458e54fb8a76c740d6f9c33bf/roles/262cff71501a4fd3b5d8663bc982bde7 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:00:53.229640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1157b2-c230-4896-8465-b8b0ed5294ca None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:53.230205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1157b2-c230-4896-8465-b8b0ed5294ca None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:53.230205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1157b2-c230-4896-8465-b8b0ed5294ca None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:53.232187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 612/1233] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:00:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:00:53.238713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e42d96-fc7c-4949-aa64-b445701f903b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:53.239307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e42d96-fc7c-4949-aa64-b445701f903b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:53.239401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e42d96-fc7c-4949-aa64-b445701f903b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:53.261810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b2e42d96-fc7c-4949-aa64-b445701f903b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:00:53.263059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b2e42d96-fc7c-4949-aa64-b445701f903b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:00:53.335802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 622/1234] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:00:53 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:00:53.360760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1157b2-c230-4896-8465-b8b0ed5294ca None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:53.387383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:53.468952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1157b2-c230-4896-8465-b8b0ed5294ca None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:53.504262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:53.504921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:53.505197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:53.505469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:53.515768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 03:00:53.520647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:53.521689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:53.522544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 613/1235] 10.222.0.42 () {56 vars in 1102 bytes} [Tue Jul 26 03:00:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:53.528147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e42d96-fc7c-4949-aa64-b445701f903b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:53.576890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:53.635037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e42d96-fc7c-4949-aa64-b445701f903b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:53.674071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:53.674748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:53.675048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:53.675353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:53.676116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] RBAC: Adding query filter params (name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:53.676509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:53.678409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:53.687477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 623/1236] 10.222.0.42 () {56 vars in 1103 bytes} [Tue Jul 26 03:00:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:53.693423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:53.795907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:53.813944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658f5f92-716f-4d71-ba4f-bcceabd00866 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:53.853616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:53.854253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:53.854530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:53.854801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:53.865468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:53.867016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:53.881808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:53.882784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:53.883750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:53.883750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:53.883750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:53.883750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:53.888676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 614/1237] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 03:00:53.889546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:54.002305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e43cafa-baf0-4329-b320-ebe6d086526c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:54.012689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:54.040605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:54.041232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:54.041578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:54.041884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:54.042732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:54.043074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:54.044699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:54.058458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 624/1238] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:54.064778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:54.184522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19d9200-a8e0-4c40-ae5d-ee3e9a7b2ff1 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:54.221184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:54.221958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:54.222268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:54.222609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] PATH_INFO: `/v3/users/placement` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:54.226466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:54.232970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:54.234585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 03:00:54.247517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:54.248516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:54.249705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 615/1239] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:54 2022] GET /identity/v3/users/placement => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:54.255501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:54.370938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f99289-156d-40bf-9ca8-946225505d63 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:54.409555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:54.410180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:54.410458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:54.410734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:54.411568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:54.411875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:54.413449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:54.432491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 625/1240] 10.222.0.42 () {56 vars in 1143 bytes} [Tue Jul 26 03:00:54 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:54.438574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:54.554575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab88136-dae8-482e-8d5e-f0950ab6658b None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:54.590332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:54.591353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:54.591697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:54.591994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:54.662861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:54.671438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:54.672997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:54.792784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 616/1241] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 03:00:54 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 26 03:00:54.799149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:54.829865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:54.910196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1882798f-3e7c-4f35-9098-419c88a833a0 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:54.950194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:54.950884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:54.951213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:54.951608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:54.963609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:54.965288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:54.983951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:54.985064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:54.985972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:54.985972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:54.985972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:54.985972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:54.985972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 626/1242] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 03:00:54.991088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:55.088941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:55.095670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73405636-9bc1-4f78-8bee-6f715660edbc None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:55.132522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:55.133153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:55.133433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:55.133719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:55.134406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:55.134706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:55.136134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:55.148001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 617/1243] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:55.154585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:55.243668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:55.269981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4044d2-44f3-42ce-ae85-7c55ee2d14ee None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:55.307220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:55.307972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:55.308336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:55.308601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:55.316139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:55.317822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 03:00:55.331466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:55.332574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:55.333694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 627/1244] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 03:00:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:55.341241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:55.456963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92c6a99-1f1f-48b7-914f-4bab8643d437 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:55.494713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c703137-126b-41e2-b570-5085975075a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:55.495478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c703137-126b-41e2-b570-5085975075a3 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:55.495662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c703137-126b-41e2-b570-5085975075a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:55.495936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c703137-126b-41e2-b570-5085975075a3 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:55.496696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c703137-126b-41e2-b570-5085975075a3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:55.497007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c703137-126b-41e2-b570-5085975075a3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:55.498792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c703137-126b-41e2-b570-5085975075a3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:55.513044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 618/1245] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 03:00:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:00:55.520189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:55.638409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb143ac-7b98-4629-a61e-2cdc63df0602 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:55.675322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52cf3e1-f457-488f-b1e8-2d24f9b3b4e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:55.676098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52cf3e1-f457-488f-b1e8-2d24f9b3b4e1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:55.676421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52cf3e1-f457-488f-b1e8-2d24f9b3b4e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:55.676694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52cf3e1-f457-488f-b1e8-2d24f9b3b4e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:55.677565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52cf3e1-f457-488f-b1e8-2d24f9b3b4e1 None admin] RBAC: Adding query filter params (role.id=262cff71501a4fd3b5d8663bc982bde7, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=35f28f9458e54fb8a76c740d6f9c33bf) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:55.677883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52cf3e1-f457-488f-b1e8-2d24f9b3b4e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:55.679368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52cf3e1-f457-488f-b1e8-2d24f9b3b4e1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:55.685982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 628/1246] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 03:00:55 2022] GET /identity/v3/role_assignments?user.id=35f28f9458e54fb8a76c740d6f9c33bf&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=262cff71501a4fd3b5d8663bc982bde7 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:57.806747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf59560-f083-41df-8512-9d625fff975d None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:57.807361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf59560-f083-41df-8512-9d625fff975d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:57.807361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf59560-f083-41df-8512-9d625fff975d None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:57.808677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 619/1247] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:00:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:00:57.814477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecf851-8155-48ed-b1ed-cb6369b8312c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:57.815051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecf851-8155-48ed-b1ed-cb6369b8312c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:57.815051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecf851-8155-48ed-b1ed-cb6369b8312c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:57.821739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:57.837558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-fcecf851-8155-48ed-b1ed-cb6369b8312c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:00:57.839358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fcecf851-8155-48ed-b1ed-cb6369b8312c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:00:57.916763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 629/1248] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:00:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 26 03:00:57.940667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf59560-f083-41df-8512-9d625fff975d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:57.960437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:58.052557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf59560-f083-41df-8512-9d625fff975d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:58.089658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:58.090230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:58.090513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:58.091216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:58.103945 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 03:00:58.109309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:58.110743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:58.111850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 620/1249] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 03:00:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:58.116976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecf851-8155-48ed-b1ed-cb6369b8312c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:58.231218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecf851-8155-48ed-b1ed-cb6369b8312c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:58.263893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:58.270073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:58.270789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:58.271151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:58.271428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:58.272157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:58.272466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:58.274015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:58.283627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 630/1250] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 03:00:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:58.290204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:58.379579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:58.409957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17e3744-3ff7-4221-b895-30b9c1bb7ead None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:58.451280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:58.452689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:58.452689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:58.452689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:58.464675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:58.466543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:58.484056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:58.485452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:58.486640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:58.486640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:58.486640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:58.486640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:58.486640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 621/1251] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:58.491877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:58.608236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c97b39f-b677-4f8b-8fd4-24b1b97d0d48 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:58.645667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:58.646248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:58.646552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:58.646828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:58.647627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:58.647943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:58.649508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:58.663391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 631/1252] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:00:58.669357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:58.793169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42df384-bde4-4682-bae2-f72886470574 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:58.825554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:58.835501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:58.836165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:58.836440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:58.836752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:58.846202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:58.847781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 03:00:58.859821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:58.860862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:00:58.862143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 622/1253] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:00:58.867374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:58.935718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:58.985866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828a661a-1411-4e88-8231-68132c925ae5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:59.029976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:59.030650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:59.030937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:59.031226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:59.032073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:59.032383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:59.033947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:59.054501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 632/1254] 10.222.0.42 () {56 vars in 1143 bytes} [Tue Jul 26 03:00:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:59.061147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:59.180884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcced970-e546-4cab-b2c2-6c99442fc7b6 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:59.220068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:59.220650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:59.220928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:59.221533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:59.285269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:59.300112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:59.301625 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:59.416573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 623/1255] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 03:00:59 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 26 03:00:59.423534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:59.535244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58bf2a1-a877-4ef0-8133-bc2c6e7cc4f5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:59.549527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:59.574131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:59.574785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:59.575101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:59.575414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:59.588116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:59.589714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:00:59.611589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:59.612978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:59.614007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:00:59.614007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:00:59.614007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:00:59.614007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:59.614007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 633/1256] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:00:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 26 03:00:59.619415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:59.718518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:00:59.734539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e1e917-6308-4147-a900-5d0a65ea1f45 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:59.787680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:59.788236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:59.788515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:59.788790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:59.789499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:00:59.789807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:59.791320 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:59.804720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 624/1257] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:00:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:00:59.810790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:00:59.934320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf10d06-31db-4f15-aadc-03cc7c2c81e5 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:00:59.971850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:00:59.972663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:00:59.972946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:00:59.973224 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:00:59.981208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:00:59.982808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 03:00:59.996465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:00:59.997661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:00:59.998643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 634/1258] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 03:00:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:00.004224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:00.116754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e02a880-711d-4b94-aa81-f3a77ffdea5c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:00.154390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60250dac-5e63-4fd6-9dda-832b24e75f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:00.155108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60250dac-5e63-4fd6-9dda-832b24e75f18 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:00.155390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60250dac-5e63-4fd6-9dda-832b24e75f18 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:00.155669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60250dac-5e63-4fd6-9dda-832b24e75f18 None admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:00.156441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60250dac-5e63-4fd6-9dda-832b24e75f18 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:00.156754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60250dac-5e63-4fd6-9dda-832b24e75f18 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:00.158247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60250dac-5e63-4fd6-9dda-832b24e75f18 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:00.171140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 625/1259] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 03:01:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:00.177614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:00.207615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:00.289827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77651877-d748-4386-b3e0-c9ca0a3f2762 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:00.334374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f074b8-8846-48b6-9799-08cdbf8c15de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:00.335089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f074b8-8846-48b6-9799-08cdbf8c15de None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:00.335374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f074b8-8846-48b6-9799-08cdbf8c15de None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:00.335651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f074b8-8846-48b6-9799-08cdbf8c15de None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:00.336508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f074b8-8846-48b6-9799-08cdbf8c15de None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=35f28f9458e54fb8a76c740d6f9c33bf) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:00.336849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f074b8-8846-48b6-9799-08cdbf8c15de None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:00.338346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f074b8-8846-48b6-9799-08cdbf8c15de None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:00.345530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 635/1260] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 03:01:00 2022] GET /identity/v3/role_assignments?user.id=35f28f9458e54fb8a76c740d6f9c33bf&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:02.411588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09808e03-68e9-4451-8401-d8d5c38d91bc None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:02.412432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09808e03-68e9-4451-8401-d8d5c38d91bc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:02.412432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09808e03-68e9-4451-8401-d8d5c38d91bc None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:02.414585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 626/1261] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:01:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:01:02.422492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cf44f3-b5fb-4a36-8b47-ba2386e7a2e2 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:02.422978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cf44f3-b5fb-4a36-8b47-ba2386e7a2e2 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:02.423072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cf44f3-b5fb-4a36-8b47-ba2386e7a2e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:02.443309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-66cf44f3-b5fb-4a36-8b47-ba2386e7a2e2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:01:02.444483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-66cf44f3-b5fb-4a36-8b47-ba2386e7a2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:01:02.516084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 636/1262] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:01:02 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:01:02.550018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09808e03-68e9-4451-8401-d8d5c38d91bc None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:02.566356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:02.674075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09808e03-68e9-4451-8401-d8d5c38d91bc None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:02.718418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:02.719460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:02.720014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:02.720215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:02.736168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 03:01:02.741911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:02.743215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:01:02.744132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 627/1263] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 03:01:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:02.749618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cf44f3-b5fb-4a36-8b47-ba2386e7a2e2 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:02.769518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:02.860219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cf44f3-b5fb-4a36-8b47-ba2386e7a2e2 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:02.897486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:02.898107 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:02.898383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:02.898655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:02.899411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:02.899712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:02.901094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:02.909371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 637/1264] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 03:01:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:02.917128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:02.988505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:03.027223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe8790e-06af-48ef-b879-b4503a06ec25 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:03.070059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:03.070710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:03.070976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:03.071324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:03.081606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:03.083106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:01:03.098034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:03.099118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:03.100087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:01:03.100087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:01:03.100087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:03.100087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:01:03.100087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 628/1265] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:01:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:03.106799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:03.188985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:03.210031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536e6a0d-dcf5-408f-89d0-a63affc5541d None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:03.244162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:03.244702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:03.244981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:03.245249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:03.245938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:03.246238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:03.247631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:03.259116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 638/1266] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:01:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:03.265365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:03.381820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4071c80b-e5cc-4cdd-9bb0-fca9d203a0d8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:03.402133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:03.421933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:03.422573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:03.422878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:03.423150 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] PATH_INFO: `/v3/users/placement` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:03.431870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:03.433334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 03:01:03.444659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:03.445696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:01:03.446798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 629/1267] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:01:03 2022] GET /identity/v3/users/placement => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:03.451852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:03.556905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d479a0-52aa-43cb-aff4-52e6b22b8f19 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:03.594409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:03.595051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:03.595332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:03.595600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:03.596394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:03.596694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:03.598114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:03.607087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:03.616159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 639/1268] 10.222.0.42 () {56 vars in 1143 bytes} [Tue Jul 26 03:01:03 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:03.623067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:03.740212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b352393a-ea0d-4431-b75a-185e66bffcaf None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:03.777890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:03.778529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:03.778810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:03.779117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:03.790099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:03.791591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:01:03.805666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:03.806686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:03.807794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:01:03.807794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:01:03.807794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:03.807794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:01:03.807794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 630/1269] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:01:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:03.812876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:03.920437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a23c447-16e3-4f30-a8d4-0d74906f724a None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:03.955429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:03.956050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:03.956323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:03.956599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:03.957283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:03.957584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:03.958981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:03.980368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 640/1270] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:01:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:03.986561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:04.028167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:04.100862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e7b086-c35a-485e-8325-6f95f2776e7f None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:04.140056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:04.140707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:04.140972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:04.141244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:04.148465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:04.149914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 03:01:04.161696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:04.162739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:01:04.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 631/1271] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 03:01:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:04.171761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:04.232050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:04.279043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b467c2-ae64-4e34-814a-63f3a78fc20e None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:04.316359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcfe3f2-e087-48a8-a1d7-9171ede9e21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:04.316976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfe3f2-e087-48a8-a1d7-9171ede9e21b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:04.317259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfe3f2-e087-48a8-a1d7-9171ede9e21b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:04.317537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcfe3f2-e087-48a8-a1d7-9171ede9e21b None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:04.318297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcfe3f2-e087-48a8-a1d7-9171ede9e21b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:04.318606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcfe3f2-e087-48a8-a1d7-9171ede9e21b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:04.320105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdcfe3f2-e087-48a8-a1d7-9171ede9e21b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:04.332567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 641/1272] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 03:01:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:04.339466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:04.441018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:04.452547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844f4cd3-ade4-4032-a0e9-6149b9bdc138 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:04.489709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3485f6-2cff-4fe5-80af-96b1821215c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:04.490370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3485f6-2cff-4fe5-80af-96b1821215c3 None admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:04.490643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3485f6-2cff-4fe5-80af-96b1821215c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:04.490916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3485f6-2cff-4fe5-80af-96b1821215c3 None admin] PATH_INFO: `/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/35f28f9458e54fb8a76c740d6f9c33bf/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:04.492851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3485f6-2cff-4fe5-80af-96b1821215c3 None admin] RBAC: Authorizing `identity:create_grant(project_id=e3a97d61573e450fa7ad485016b1f019, user_id=35f28f9458e54fb8a76c740d6f9c33bf, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:04.494284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3485f6-2cff-4fe5-80af-96b1821215c3 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:04.508889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 632/1273] 10.222.0.42 () {58 vars in 1439 bytes} [Tue Jul 26 03:01:04 2022] PUT /identity/v3/projects/e3a97d61573e450fa7ad485016b1f019/users/35f28f9458e54fb8a76c740d6f9c33bf/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:01:06.467147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c15cf67-bf71-4bd5-9076-11730f6fdab4 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:06.467773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c15cf67-bf71-4bd5-9076-11730f6fdab4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:06.467773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c15cf67-bf71-4bd5-9076-11730f6fdab4 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:06.469083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 642/1274] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:01:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:01:06.475553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b124f4-5b1a-4b74-af1c-994b1688060b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:06.476297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b124f4-5b1a-4b74-af1c-994b1688060b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:06.476436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b124f4-5b1a-4b74-af1c-994b1688060b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:06.498400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-58b124f4-5b1a-4b74-af1c-994b1688060b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:01:06.499672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-58b124f4-5b1a-4b74-af1c-994b1688060b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:01:06.572849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 633/1275] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:01:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:01:06.604091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c15cf67-bf71-4bd5-9076-11730f6fdab4 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:06.715443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c15cf67-bf71-4bd5-9076-11730f6fdab4 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:06.740771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:06.762460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:06.763141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:06.763425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:06.763700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:06.778414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 26 03:01:06.783499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:06.784611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:06.785701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 643/1276] 10.222.0.42 () {56 vars in 1096 bytes} [Tue Jul 26 03:01:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:06.792390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b124f4-5b1a-4b74-af1c-994b1688060b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:06.880950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:06.900487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b124f4-5b1a-4b74-af1c-994b1688060b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:06.934818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:06.935461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:06.935735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:06.936004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:06.936847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:06.937147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:06.938527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:06.946507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 634/1277] 10.222.0.42 () {56 vars in 1099 bytes} [Tue Jul 26 03:01:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:06.954741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:07.080137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a03d1-3163-4ded-9e63-19878dfc42b4 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:07.128603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:07.129266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:07.129564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:07.129778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:07.141160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:07.142619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:01:07.158481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:07.159572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:07.160881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:01:07.160881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:01:07.160881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:07.160881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:07.160881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 644/1278] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:01:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:07.166925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:07.272677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c1320-a41a-447c-824e-8960db1d3865 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:07.311613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:07.312229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:07.312504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:07.312771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:07.313463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:07.313764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:07.315163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:07.326984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 635/1279] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:01:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:07.333681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:07.370797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:07.451580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175127de-b165-4a51-bbd5-852faa28757b None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:07.504160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:07.504897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:07.505040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:07.505325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] PATH_INFO: `/v3/users/placement` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:07.516760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:07.519024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 26 03:01:07.534665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:07.535687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:07.536729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 645/1280] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:01:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:07.542221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:07.547193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:07.653434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5cf5c-3cdd-4208-8ad9-210b088795c2 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:07.692230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:07.692849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:07.693123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:07.693392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:07.694191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:07.694498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:07.695984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:07.713688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 636/1281] 10.222.0.42 () {56 vars in 1143 bytes} [Tue Jul 26 03:01:07 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:07.720521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:07.822493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:07.833158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec9aa1-bc59-4f96-940d-a652cc38915c None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:07.868997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:07.869614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:07.869890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:07.870161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:07.943074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:07.944522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:08.047101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:08.058579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 646/1282] 10.222.0.42 () {58 vars in 1282 bytes} [Tue Jul 26 03:01:07 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 26 03:01:08.064984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:08.131180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:08.173115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a932eb1-2210-4737-a0e6-7893e631ace8 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:08.207931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:08.208566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:08.208841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:08.209117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:08.220591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:08.222051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 26 03:01:08.237315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:08.238288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:08.239254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 26 03:01:08.239254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 26 03:01:08.239254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 26 03:01:08.239254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:01:08.239254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 637/1283] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:01:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:08.247535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:08.370613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee06d03-0b2d-4080-80f4-4443c6e40605 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:08.406743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:08.407316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:08.407590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:08.407865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:08.408574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:08.408880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:08.410270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:08.422989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 647/1284] 10.222.0.42 () {56 vars in 1109 bytes} [Tue Jul 26 03:01:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:08.429216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:08.543501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f9ae6-ef2c-4989-9648-e0a931900b9c None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:08.559722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:08.587822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:08.588460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:08.588760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:08.589025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:08.596107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:08.597559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 26 03:01:08.609887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:08.610907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: ERROR keystone.server.flask.application Jul 26 03:01:08.611887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 638/1285] 10.222.0.42 () {56 vars in 1111 bytes} [Tue Jul 26 03:01:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:08.619342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:08.680141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:08.739157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cfa11a-7472-4945-8788-d5a1abb3f8f1 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:08.773986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3da36e-463e-45c6-98db-17b9cfde280c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:08.774596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3da36e-463e-45c6-98db-17b9cfde280c None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:08.774868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3da36e-463e-45c6-98db-17b9cfde280c None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:08.775173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3da36e-463e-45c6-98db-17b9cfde280c None admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:08.775925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3da36e-463e-45c6-98db-17b9cfde280c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:08.776232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3da36e-463e-45c6-98db-17b9cfde280c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:08.777661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3da36e-463e-45c6-98db-17b9cfde280c None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:08.789527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 648/1286] 10.222.0.42 () {56 vars in 1148 bytes} [Tue Jul 26 03:01:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:08.798478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:08.917486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92043663-723d-4f0a-bef7-6476c4caff05 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:08.953601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b75623e-e79b-411f-9a3d-ee49782fb7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:08.954141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b75623e-e79b-411f-9a3d-ee49782fb7a8 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:08.954414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b75623e-e79b-411f-9a3d-ee49782fb7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:08.954681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b75623e-e79b-411f-9a3d-ee49782fb7a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:08.955495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b75623e-e79b-411f-9a3d-ee49782fb7a8 None admin] RBAC: Adding query filter params (role.id=50e7a7f2795f407e8da49801c56f6c5b, scope.project.id=e3a97d61573e450fa7ad485016b1f019, user.id=35f28f9458e54fb8a76c740d6f9c33bf) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:08.955802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b75623e-e79b-411f-9a3d-ee49782fb7a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:08.957263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b75623e-e79b-411f-9a3d-ee49782fb7a8 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:08.963187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 639/1287] 10.222.0.42 () {56 vars in 1374 bytes} [Tue Jul 26 03:01:08 2022] GET /identity/v3/role_assignments?user.id=35f28f9458e54fb8a76c740d6f9c33bf&scope.project.id=e3a97d61573e450fa7ad485016b1f019&role.id=50e7a7f2795f407e8da49801c56f6c5b => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:10.958428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf78a81-25a4-42d9-bc2a-5c18399643ea None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:10.959376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf78a81-25a4-42d9-bc2a-5c18399643ea None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:10.959376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf78a81-25a4-42d9-bc2a-5c18399643ea None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:10.961550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 649/1288] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:01:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:01:10.967353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed32a98-34c0-4e0e-a163-71d132c943f7 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:10.967803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed32a98-34c0-4e0e-a163-71d132c943f7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:10.967910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed32a98-34c0-4e0e-a163-71d132c943f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:10.987985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1ed32a98-34c0-4e0e-a163-71d132c943f7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:01:10.989164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1ed32a98-34c0-4e0e-a163-71d132c943f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:01:11.020890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:11.060387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 640/1289] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:01:10 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:01:11.082455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf78a81-25a4-42d9-bc2a-5c18399643ea None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:11.201429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf78a81-25a4-42d9-bc2a-5c18399643ea None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:11.213902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:11.241495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:11.242135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:11.242412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:11.242660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:11.243353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:11.244721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:11.254525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:11.255778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:11.256647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 650/1290] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 03:01:11 2022] GET /identity/v3/services/placement => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:11.262169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed32a98-34c0-4e0e-a163-71d132c943f7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:11.363195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed32a98-34c0-4e0e-a163-71d132c943f7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:11.398976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8869eef9-4744-4fdb-ab34-3e2670ef481f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:11.399594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8869eef9-4744-4fdb-ab34-3e2670ef481f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:11.399869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8869eef9-4744-4fdb-ab34-3e2670ef481f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:11.400142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8869eef9-4744-4fdb-ab34-3e2670ef481f None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:11.400814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8869eef9-4744-4fdb-ab34-3e2670ef481f None admin] RBAC: Adding query filter params (name=placement) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:11.401122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8869eef9-4744-4fdb-ab34-3e2670ef481f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:11.402495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8869eef9-4744-4fdb-ab34-3e2670ef481f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:11.407663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 641/1291] 10.222.0.42 () {56 vars in 1116 bytes} [Tue Jul 26 03:01:11 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:11.413815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:11.523450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b0b595-cc9e-4cca-90a2-1007b1eb3d17 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:11.559195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101734a2-b1f0-4c19-bd48-2480c7dd3a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:11.559810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101734a2-b1f0-4c19-bd48-2480c7dd3a0a None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:11.560085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101734a2-b1f0-4c19-bd48-2480c7dd3a0a None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:11.560352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101734a2-b1f0-4c19-bd48-2480c7dd3a0a None admin] PATH_INFO: `/v3/services` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:11.561033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101734a2-b1f0-4c19-bd48-2480c7dd3a0a None admin] RBAC: Adding query filter params (type=placement) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:11.561357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101734a2-b1f0-4c19-bd48-2480c7dd3a0a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:11.562726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101734a2-b1f0-4c19-bd48-2480c7dd3a0a None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:11.567269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 651/1292] 10.222.0.42 () {56 vars in 1116 bytes} [Tue Jul 26 03:01:11 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:13.546804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8a105-ff6c-4b59-bde0-aabe53ecae3d None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:13.547720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8a105-ff6c-4b59-bde0-aabe53ecae3d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:13.547895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8a105-ff6c-4b59-bde0-aabe53ecae3d None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:13.550084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 642/1293] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:01:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:01:13.556389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec0c9da-4ffe-4ea1-b227-aa8a7567d39b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:13.557114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec0c9da-4ffe-4ea1-b227-aa8a7567d39b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:13.557261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec0c9da-4ffe-4ea1-b227-aa8a7567d39b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:13.579263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-bec0c9da-4ffe-4ea1-b227-aa8a7567d39b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:01:13.580462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bec0c9da-4ffe-4ea1-b227-aa8a7567d39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:01:13.612213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:13.651825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 652/1294] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:01:13 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:01:13.687791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda8a105-ff6c-4b59-bde0-aabe53ecae3d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:13.725228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:13.809184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda8a105-ff6c-4b59-bde0-aabe53ecae3d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:13.842821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe513da-f64a-4c0c-8a72-b3b8ff3b5f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:13.843471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe513da-f64a-4c0c-8a72-b3b8ff3b5f32 None admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:13.843750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe513da-f64a-4c0c-8a72-b3b8ff3b5f32 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:13.844024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe513da-f64a-4c0c-8a72-b3b8ff3b5f32 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:13.844719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe513da-f64a-4c0c-8a72-b3b8ff3b5f32 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:13.846101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe513da-f64a-4c0c-8a72-b3b8ff3b5f32 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:13.864139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 643/1295] 10.222.0.42 () {60 vars in 1141 bytes} [Tue Jul 26 03:01:13 2022] POST /identity/v3/services => generated 246 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:01:15.837587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da50df87-5a37-4b9e-b358-e6d247afa8a7 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:15.838538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da50df87-5a37-4b9e-b358-e6d247afa8a7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:15.838538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da50df87-5a37-4b9e-b358-e6d247afa8a7 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:15.841168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 653/1296] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:01:15 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:01:15.847555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e159a03-9f8c-44ac-b7f9-ceb1181997e5 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:15.847986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e159a03-9f8c-44ac-b7f9-ceb1181997e5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:15.848088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e159a03-9f8c-44ac-b7f9-ceb1181997e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:15.867498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8e159a03-9f8c-44ac-b7f9-ceb1181997e5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:01:15.868705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8e159a03-9f8c-44ac-b7f9-ceb1181997e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:01:15.949979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 644/1297] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:01:15 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:01:15.973939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da50df87-5a37-4b9e-b358-e6d247afa8a7 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:16.094591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da50df87-5a37-4b9e-b358-e6d247afa8a7 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:16.136131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:16.136776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:16.137044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:16.137323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] PATH_INFO: `/v3/services/placement` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:16.137948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:16.139433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:16.148642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:16.149793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:16.150676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 654/1298] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 03:01:15 2022] GET /identity/v3/services/placement => generated 90 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:16.156535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e159a03-9f8c-44ac-b7f9-ceb1181997e5 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:16.178458 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:16.272622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e159a03-9f8c-44ac-b7f9-ceb1181997e5 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:16.310093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:16.310711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:16.310989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:16.311365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:16.312037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:16.312337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:16.313708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:16.319633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 645/1299] 10.222.0.42 () {56 vars in 1116 bytes} [Tue Jul 26 03:01:16 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:16.328896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:16.363072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:16.434234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d945ff5f-38e6-4499-977d-db057ed30bda None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:16.469904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24702b45-edd5-437a-b3c4-122e3ac78009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:16.470515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24702b45-edd5-437a-b3c4-122e3ac78009 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:16.470793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24702b45-edd5-437a-b3c4-122e3ac78009 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:16.471094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24702b45-edd5-437a-b3c4-122e3ac78009 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:16.471865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24702b45-edd5-437a-b3c4-122e3ac78009 None admin] RBAC: Adding query filter params (interface=public, service_id=896b8ca8c31e44709c417412062d9b83, region_id=RegionOne) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:16.472179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24702b45-edd5-437a-b3c4-122e3ac78009 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:16.473565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24702b45-edd5-437a-b3c4-122e3ac78009 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:16.478711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 655/1300] 10.222.0.42 () {56 vars in 1251 bytes} [Tue Jul 26 03:01:16 2022] GET /identity/v3/endpoints?service_id=896b8ca8c31e44709c417412062d9b83&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:01:16.489180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:16.588390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:16.607471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf28158-80f9-4028-8bb0-e71f0aaf9e89 None admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:16.642748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e8b66b-d5ec-42cb-9f60-6dd0cac55fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:16.643314 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e8b66b-d5ec-42cb-9f60-6dd0cac55fbd None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:16.643591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e8b66b-d5ec-42cb-9f60-6dd0cac55fbd None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:16.643853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e8b66b-d5ec-42cb-9f60-6dd0cac55fbd None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:16.644499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e8b66b-d5ec-42cb-9f60-6dd0cac55fbd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:16.645869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e8b66b-d5ec-42cb-9f60-6dd0cac55fbd None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:16.653151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 646/1301] 10.222.0.42 () {56 vars in 1087 bytes} [Tue Jul 26 03:01:16 2022] GET /identity/v3/services => generated 2191 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:01:18.666095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a5b1c-2a2b-471c-af37-23fc9cdf74a9 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:18.666699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a5b1c-2a2b-471c-af37-23fc9cdf74a9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:18.666699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a5b1c-2a2b-471c-af37-23fc9cdf74a9 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:18.668014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 656/1302] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:01:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:01:18.674337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff12255-c9c4-4700-a551-4169fcc74b1a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:18.674776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff12255-c9c4-4700-a551-4169fcc74b1a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:18.674889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff12255-c9c4-4700-a551-4169fcc74b1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:18.697952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-bff12255-c9c4-4700-a551-4169fcc74b1a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:01:18.699441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bff12255-c9c4-4700-a551-4169fcc74b1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:01:18.775900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 647/1303] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:01:18 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 26 03:01:18.799832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1a5b1c-2a2b-471c-af37-23fc9cdf74a9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:18.902368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:18.912581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1a5b1c-2a2b-471c-af37-23fc9cdf74a9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:18.958081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:18.958655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:18.959077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:18.959241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:18.959881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:18.961272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 26 03:01:18.971664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 26 03:01:18.974580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.get_or_create( Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application with Lock( Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self._enter() Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application return self.creator() Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application created_value = creator( Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: ERROR keystone.server.flask.application Jul 26 03:01:18.976477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 657/1304] 10.222.0.42 () {56 vars in 1117 bytes} [Tue Jul 26 03:01:18 2022] GET /identity/v3/services/placement => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 26 03:01:18.982279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff12255-c9c4-4700-a551-4169fcc74b1a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:19.062397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:19.087562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff12255-c9c4-4700-a551-4169fcc74b1a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:19.127952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c69a3-0b1b-4fe9-a24f-b8d3b86a218f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:19.128568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c69a3-0b1b-4fe9-a24f-b8d3b86a218f None admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:19.128839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c69a3-0b1b-4fe9-a24f-b8d3b86a218f None admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:19.129110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c69a3-0b1b-4fe9-a24f-b8d3b86a218f None admin] PATH_INFO: `/v3/services` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:19.129789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c69a3-0b1b-4fe9-a24f-b8d3b86a218f None admin] RBAC: Adding query filter params (name=placement) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:01:19.130091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c69a3-0b1b-4fe9-a24f-b8d3b86a218f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:19.131512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c69a3-0b1b-4fe9-a24f-b8d3b86a218f None admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:19.137159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 648/1305] 10.222.0.42 () {56 vars in 1116 bytes} [Tue Jul 26 03:01:18 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:01:19.143694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:01:19.252335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa121587-7802-4a3b-aa97-8ba070089d06 None admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:01:19.289617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db1a833-99f2-4180-b745-c9e8aaa86103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:01:19.290235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db1a833-99f2-4180-b745-c9e8aaa86103 None admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:01:19.290511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db1a833-99f2-4180-b745-c9e8aaa86103 None admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:01:19.290782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db1a833-99f2-4180-b745-c9e8aaa86103 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:01:19.291521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db1a833-99f2-4180-b745-c9e8aaa86103 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:01:19.292898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db1a833-99f2-4180-b745-c9e8aaa86103 None admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:01:19.316467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 658/1306] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:01:19 2022] POST /identity/v3/endpoints => generated 331 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:01:29.133875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=30 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:01:29.312157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:22.603935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921cde22-3e0c-400f-b11b-22943f024a0b None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:22.604977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921cde22-3e0c-400f-b11b-22943f024a0b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:22.605468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921cde22-3e0c-400f-b11b-22943f024a0b None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:22.612440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c5e508-dd74-4994-802c-d6b5cfb07fd4 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:22.613345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c5e508-dd74-4994-802c-d6b5cfb07fd4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:22.613821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c5e508-dd74-4994-802c-d6b5cfb07fd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:22.618610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 649/1307] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:22 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:22.703508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-34c5e508-dd74-4994-802c-d6b5cfb07fd4 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:22.705256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-34c5e508-dd74-4994-802c-d6b5cfb07fd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:22.874480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 659/1308] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:22.925062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab15b6-c41e-4c66-bff8-db309c5e9c3b None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:22.925397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab15b6-c41e-4c66-bff8-db309c5e9c3b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:22.925763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab15b6-c41e-4c66-bff8-db309c5e9c3b None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:22.927300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 650/1309] 10.222.0.42 () {54 vars in 965 bytes} [Tue Jul 26 03:02:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:22.935702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b7e432-a760-442b-a65c-9784c314ae65 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:22.937242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b7e432-a760-442b-a65c-9784c314ae65 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:22.937630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b7e432-a760-442b-a65c-9784c314ae65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:23.038473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-71b7e432-a760-442b-a65c-9784c314ae65 None None] MFA Rules not processed for user `29a1868eb95f4396aa3fd2da677b40f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:23.039906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-71b7e432-a760-442b-a65c-9784c314ae65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:23.077905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:23.151323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 660/1310] 10.222.0.42 () {58 vars in 1062 bytes} [Tue Jul 26 03:02:22 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:23.154868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ab15b6-c41e-4c66-bff8-db309c5e9c3b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:23.156770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e9ab15b6-c41e-4c66-bff8-db309c5e9c3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:23.270806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ab15b6-c41e-4c66-bff8-db309c5e9c3b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:23.348683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ddae40-f9ea-4741-96d8-50bac736d7c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:23.349367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ddae40-f9ea-4741-96d8-50bac736d7c8 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:23.349646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ddae40-f9ea-4741-96d8-50bac736d7c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:23.349921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ddae40-f9ea-4741-96d8-50bac736d7c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:23.351731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a6ddae40-f9ea-4741-96d8-50bac736d7c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:23.413457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:23.473124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ddae40-f9ea-4741-96d8-50bac736d7c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:23.474577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ddae40-f9ea-4741-96d8-50bac736d7c8 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:23.494899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 651/1311] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:25.804633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb8da06-526d-4a16-950d-4e6f2e98339e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:25.805211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb8da06-526d-4a16-950d-4e6f2e98339e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:25.805211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb8da06-526d-4a16-950d-4e6f2e98339e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:25.806580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 661/1312] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:25.812246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d32a558-2602-4a0e-81a1-7052006de993 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:25.812688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d32a558-2602-4a0e-81a1-7052006de993 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:25.813073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d32a558-2602-4a0e-81a1-7052006de993 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:25.887576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3d32a558-2602-4a0e-81a1-7052006de993 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:25.888803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3d32a558-2602-4a0e-81a1-7052006de993 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:26.011275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 652/1313] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:26.049927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb8da06-526d-4a16-950d-4e6f2e98339e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:26.051712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ccb8da06-526d-4a16-950d-4e6f2e98339e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:26.171256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb8da06-526d-4a16-950d-4e6f2e98339e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:26.255585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:26.258633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abe1241-1c58-4da5-866b-61c0f8bccc4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:26.259420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abe1241-1c58-4da5-866b-61c0f8bccc4c service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:26.259941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abe1241-1c58-4da5-866b-61c0f8bccc4c service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:26.260331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abe1241-1c58-4da5-866b-61c0f8bccc4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:26.262739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9abe1241-1c58-4da5-866b-61c0f8bccc4c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:26.381379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abe1241-1c58-4da5-866b-61c0f8bccc4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:26.382838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abe1241-1c58-4da5-866b-61c0f8bccc4c service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:26.401540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 662/1314] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:28.729493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a821666e-dfa8-44ef-8f0e-7725215ad35a None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:28.730355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a821666e-dfa8-44ef-8f0e-7725215ad35a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:28.730477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a821666e-dfa8-44ef-8f0e-7725215ad35a None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:28.732651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 653/1315] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:28.738501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ef04c-465e-4480-8a73-a24403342ad3 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:28.738967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ef04c-465e-4480-8a73-a24403342ad3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:28.739084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ef04c-465e-4480-8a73-a24403342ad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:28.806791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-748ef04c-465e-4480-8a73-a24403342ad3 None None] MFA Rules not processed for user `f7f19a1e144b461c9f801f3d04a0e422`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:28.808028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-748ef04c-465e-4480-8a73-a24403342ad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:28.866608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:28.917239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 663/1316] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:28 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:02:28.972484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daae259-e825-4be8-a917-1d6e99910aee None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:28.972750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daae259-e825-4be8-a917-1d6e99910aee None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:28.973028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daae259-e825-4be8-a917-1d6e99910aee None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:28.975178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 654/1317] 10.222.0.42 () {54 vars in 965 bytes} [Tue Jul 26 03:02:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:28.982987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc30f56-f24a-4e25-84d9-d25c1c8afe0f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:28.983303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc30f56-f24a-4e25-84d9-d25c1c8afe0f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:28.983548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc30f56-f24a-4e25-84d9-d25c1c8afe0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:29.070112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-2bc30f56-f24a-4e25-84d9-d25c1c8afe0f None None] MFA Rules not processed for user `29a1868eb95f4396aa3fd2da677b40f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:29.071379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2bc30f56-f24a-4e25-84d9-d25c1c8afe0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:29.152960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 664/1318] 10.222.0.42 () {58 vars in 1062 bytes} [Tue Jul 26 03:02:28 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:29.162501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daae259-e825-4be8-a917-1d6e99910aee None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:29.164069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9daae259-e825-4be8-a917-1d6e99910aee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:29.175225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:29.283313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daae259-e825-4be8-a917-1d6e99910aee None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:29.361058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e31b0e2-dd20-403f-88cb-a2524293ba31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:29.361676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e31b0e2-dd20-403f-88cb-a2524293ba31 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:29.361949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e31b0e2-dd20-403f-88cb-a2524293ba31 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:29.362220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e31b0e2-dd20-403f-88cb-a2524293ba31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:29.363846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0e31b0e2-dd20-403f-88cb-a2524293ba31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:29.430351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:29.484092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e31b0e2-dd20-403f-88cb-a2524293ba31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:29.485539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e31b0e2-dd20-403f-88cb-a2524293ba31 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:29.505818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 655/1319] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:29 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:32.454131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a84ffb-6c83-4107-ad34-6cf6e3a19c42 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:32.454706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a84ffb-6c83-4107-ad34-6cf6e3a19c42 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:32.454706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a84ffb-6c83-4107-ad34-6cf6e3a19c42 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:32.456067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 665/1320] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:32.462281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbe326-452f-4f0c-8941-07e2315e591f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:32.463024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbe326-452f-4f0c-8941-07e2315e591f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:32.463177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fbe326-452f-4f0c-8941-07e2315e591f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:32.539205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-11fbe326-452f-4f0c-8941-07e2315e591f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:32.540439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-11fbe326-452f-4f0c-8941-07e2315e591f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:32.647844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 656/1321] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:32 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:32.689538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a84ffb-6c83-4107-ad34-6cf6e3a19c42 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:32.692049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-05a84ffb-6c83-4107-ad34-6cf6e3a19c42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:32.752027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:32.808346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a84ffb-6c83-4107-ad34-6cf6e3a19c42 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:32.904471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1650ce7-8776-4ae4-88ef-492d38c32898 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:32.905009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1650ce7-8776-4ae4-88ef-492d38c32898 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:32.905285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1650ce7-8776-4ae4-88ef-492d38c32898 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:32.905577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1650ce7-8776-4ae4-88ef-492d38c32898 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:32.907239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b1650ce7-8776-4ae4-88ef-492d38c32898 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:33.014065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:33.032522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1650ce7-8776-4ae4-88ef-492d38c32898 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:33.033969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1650ce7-8776-4ae4-88ef-492d38c32898 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:33.055621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 666/1322] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:32 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:35.782392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cc5791-4d1b-4578-bf11-163d15f735a4 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:35.782983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cc5791-4d1b-4578-bf11-163d15f735a4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:35.783162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cc5791-4d1b-4578-bf11-163d15f735a4 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:35.785423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 657/1323] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:35.790912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b79efea-b35e-44c8-879d-770b7a36e097 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:35.791422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b79efea-b35e-44c8-879d-770b7a36e097 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:35.791510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b79efea-b35e-44c8-879d-770b7a36e097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:35.860927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0b79efea-b35e-44c8-879d-770b7a36e097 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:35.862080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0b79efea-b35e-44c8-879d-770b7a36e097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:35.969215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 667/1324] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:36.006028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cc5791-4d1b-4578-bf11-163d15f735a4 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:36.007608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-00cc5791-4d1b-4578-bf11-163d15f735a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:36.030791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:36.129522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cc5791-4d1b-4578-bf11-163d15f735a4 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:36.205948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c97de5-6ac4-4899-a6d6-1b1d3600504a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:36.206575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c97de5-6ac4-4899-a6d6-1b1d3600504a service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:36.206853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c97de5-6ac4-4899-a6d6-1b1d3600504a service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:36.207136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c97de5-6ac4-4899-a6d6-1b1d3600504a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:36.208789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e1c97de5-6ac4-4899-a6d6-1b1d3600504a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:36.282143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:36.329175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c97de5-6ac4-4899-a6d6-1b1d3600504a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:36.330542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c97de5-6ac4-4899-a6d6-1b1d3600504a service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:36.349963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 658/1325] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:38.390796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdcc11c-b874-47a0-863c-5dad39db0174 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:38.391568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdcc11c-b874-47a0-863c-5dad39db0174 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:38.391568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdcc11c-b874-47a0-863c-5dad39db0174 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:38.392728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 668/1326] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:38.398175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c749d5-42a6-4bd8-a118-d94f3c088720 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:38.398664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c749d5-42a6-4bd8-a118-d94f3c088720 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:38.398767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c749d5-42a6-4bd8-a118-d94f3c088720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:38.473802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-61c749d5-42a6-4bd8-a118-d94f3c088720 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:38.475082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-61c749d5-42a6-4bd8-a118-d94f3c088720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:38.576388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 659/1327] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:38.612964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdcc11c-b874-47a0-863c-5dad39db0174 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:38.614595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0fdcc11c-b874-47a0-863c-5dad39db0174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:38.677459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:38.725192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdcc11c-b874-47a0-863c-5dad39db0174 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:38.811256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5ac4d9-7601-4c47-a63f-1dae4ff1bbb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:38.811927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5ac4d9-7601-4c47-a63f-1dae4ff1bbb3 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:38.812206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5ac4d9-7601-4c47-a63f-1dae4ff1bbb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:38.812596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5ac4d9-7601-4c47-a63f-1dae4ff1bbb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:38.814247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1a5ac4d9-7601-4c47-a63f-1dae4ff1bbb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:38.927390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:38.930000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5ac4d9-7601-4c47-a63f-1dae4ff1bbb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:38.931808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5ac4d9-7601-4c47-a63f-1dae4ff1bbb3 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:38.950488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 669/1328] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:40.981679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ba92e4-8332-4bf4-b089-a14f4624466d None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:40.982240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ba92e4-8332-4bf4-b089-a14f4624466d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:40.982240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ba92e4-8332-4bf4-b089-a14f4624466d None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:40.983864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 660/1329] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:40.989024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ca357-a6b9-4c51-8065-cef70d164320 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:40.989459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ca357-a6b9-4c51-8065-cef70d164320 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:40.989546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ca357-a6b9-4c51-8065-cef70d164320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:41.060148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-6d0ca357-a6b9-4c51-8065-cef70d164320 None None] MFA Rules not processed for user `f7f19a1e144b461c9f801f3d04a0e422`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:41.061359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6d0ca357-a6b9-4c51-8065-cef70d164320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:41.166508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 670/1330] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:40 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:41.204586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ba92e4-8332-4bf4-b089-a14f4624466d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:41.206254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-08ba92e4-8332-4bf4-b089-a14f4624466d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:41.241359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:41.323062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ba92e4-8332-4bf4-b089-a14f4624466d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:41.401375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc239a4d-a035-4f00-940d-81d4ffcdc1a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:41.401909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc239a4d-a035-4f00-940d-81d4ffcdc1a8 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:41.402183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc239a4d-a035-4f00-940d-81d4ffcdc1a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:41.402448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc239a4d-a035-4f00-940d-81d4ffcdc1a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:41.404102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fc239a4d-a035-4f00-940d-81d4ffcdc1a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:41.485440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:41.518177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc239a4d-a035-4f00-940d-81d4ffcdc1a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:41.519696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc239a4d-a035-4f00-940d-81d4ffcdc1a8 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:41.540592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 661/1331] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:41 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:43.802852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a392bd-a842-45dc-9457-20cbdeb6ec47 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:43.803466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a392bd-a842-45dc-9457-20cbdeb6ec47 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:43.803466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a392bd-a842-45dc-9457-20cbdeb6ec47 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:43.804809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 671/1332] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:43.811174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1248e35-ce88-4b2b-b665-2fc4e8bfefb5 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:43.811890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1248e35-ce88-4b2b-b665-2fc4e8bfefb5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:43.812062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1248e35-ce88-4b2b-b665-2fc4e8bfefb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:43.886817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c1248e35-ce88-4b2b-b665-2fc4e8bfefb5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:43.888051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c1248e35-ce88-4b2b-b665-2fc4e8bfefb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:43.990645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 662/1333] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:44.034423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a392bd-a842-45dc-9457-20cbdeb6ec47 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:44.036006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-04a392bd-a842-45dc-9457-20cbdeb6ec47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:44.115475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:44.155608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a392bd-a842-45dc-9457-20cbdeb6ec47 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:44.235186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dfa997-4662-4546-bbe4-66a3a570f892 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:44.235806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dfa997-4662-4546-bbe4-66a3a570f892 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:44.236076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dfa997-4662-4546-bbe4-66a3a570f892 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:44.236460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dfa997-4662-4546-bbe4-66a3a570f892 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:44.238075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-44dfa997-4662-4546-bbe4-66a3a570f892 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:44.357431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dfa997-4662-4546-bbe4-66a3a570f892 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:44.359459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dfa997-4662-4546-bbe4-66a3a570f892 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:44.371219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:44.380565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 672/1334] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:46.435790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74ec35-b3b3-4220-804c-316d692d383e None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:46.436382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74ec35-b3b3-4220-804c-316d692d383e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:46.436382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74ec35-b3b3-4220-804c-316d692d383e None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:46.437746 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 663/1335] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:46.443465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49db513-b5af-4438-8ada-ed86673b3f42 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:46.443875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49db513-b5af-4438-8ada-ed86673b3f42 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:46.444079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49db513-b5af-4438-8ada-ed86673b3f42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:46.511738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f49db513-b5af-4438-8ada-ed86673b3f42 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:46.512962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f49db513-b5af-4438-8ada-ed86673b3f42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:46.616102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 673/1336] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:46.661759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba74ec35-b3b3-4220-804c-316d692d383e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:46.663387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ba74ec35-b3b3-4220-804c-316d692d383e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:46.705025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:46.772085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba74ec35-b3b3-4220-804c-316d692d383e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:46.848786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98289554-a962-4241-913a-50279bc3ba26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:46.849418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98289554-a962-4241-913a-50279bc3ba26 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:46.849694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98289554-a962-4241-913a-50279bc3ba26 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:46.849964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98289554-a962-4241-913a-50279bc3ba26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:46.851646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-98289554-a962-4241-913a-50279bc3ba26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:46.940211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:46.966728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98289554-a962-4241-913a-50279bc3ba26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:46.968445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98289554-a962-4241-913a-50279bc3ba26 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:46.990048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 664/1337] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:49.372168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253e05c-f140-4b37-987b-ce9c370bd979 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:49.372751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253e05c-f140-4b37-987b-ce9c370bd979 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:49.372751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3253e05c-f140-4b37-987b-ce9c370bd979 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:49.374147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 674/1338] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:49.379637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755d5d8-4006-4872-b61f-0fe5e0755825 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:49.380114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755d5d8-4006-4872-b61f-0fe5e0755825 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:49.380221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755d5d8-4006-4872-b61f-0fe5e0755825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:49.450258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8755d5d8-4006-4872-b61f-0fe5e0755825 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:49.451510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8755d5d8-4006-4872-b61f-0fe5e0755825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:49.552456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 665/1339] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:49 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:49.589226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253e05c-f140-4b37-987b-ce9c370bd979 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:49.590775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3253e05c-f140-4b37-987b-ce9c370bd979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:49.670260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:49.704156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3253e05c-f140-4b37-987b-ce9c370bd979 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:49.791751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9785c932-ec2d-468b-81c2-56ab15db46d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:49.792378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9785c932-ec2d-468b-81c2-56ab15db46d6 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:49.792657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9785c932-ec2d-468b-81c2-56ab15db46d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:49.792927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9785c932-ec2d-468b-81c2-56ab15db46d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:49.794585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9785c932-ec2d-468b-81c2-56ab15db46d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:49.928505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9785c932-ec2d-468b-81c2-56ab15db46d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:49.929855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9785c932-ec2d-468b-81c2-56ab15db46d6 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:49.948021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 675/1340] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:49 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:53.961975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713e11c5-b55b-4c7b-bca9-261f918184c8 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:53.961975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713e11c5-b55b-4c7b-bca9-261f918184c8 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:53.962646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713e11c5-b55b-4c7b-bca9-261f918184c8 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:53.964059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 666/1341] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:53.969473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c5ab16-b3a3-4302-b46b-34de40481be0 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:53.969950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c5ab16-b3a3-4302-b46b-34de40481be0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:53.970033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c5ab16-b3a3-4302-b46b-34de40481be0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:53.975689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:54.043685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d1c5ab16-b3a3-4302-b46b-34de40481be0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:54.044909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d1c5ab16-b3a3-4302-b46b-34de40481be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:54.152507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 676/1342] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:54.192014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713e11c5-b55b-4c7b-bca9-261f918184c8 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:54.193685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-713e11c5-b55b-4c7b-bca9-261f918184c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:54.246375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:54.306246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713e11c5-b55b-4c7b-bca9-261f918184c8 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:54.390452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f1d00f-f5be-4cf2-9e71-9f095af0013d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:54.391108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f1d00f-f5be-4cf2-9e71-9f095af0013d service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:54.391389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f1d00f-f5be-4cf2-9e71-9f095af0013d service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:54.391658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f1d00f-f5be-4cf2-9e71-9f095af0013d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:54.393377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-12f1d00f-f5be-4cf2-9e71-9f095af0013d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:54.505532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:54.522294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f1d00f-f5be-4cf2-9e71-9f095af0013d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:54.523950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f1d00f-f5be-4cf2-9e71-9f095af0013d service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:54.544325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 667/1343] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:02:57.415011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ab092-89ea-4e9d-a676-5d0b593eea82 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:57.415555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ab092-89ea-4e9d-a676-5d0b593eea82 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:57.415555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ab092-89ea-4e9d-a676-5d0b593eea82 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:57.416936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 677/1344] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:02:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:02:57.422595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b27027-e5cc-49a6-9b99-da3f8742a8a3 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:57.423036 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b27027-e5cc-49a6-9b99-da3f8742a8a3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:57.423355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b27027-e5cc-49a6-9b99-da3f8742a8a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:57.494719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-02b27027-e5cc-49a6-9b99-da3f8742a8a3 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:02:57.495982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-02b27027-e5cc-49a6-9b99-da3f8742a8a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:57.608539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 668/1345] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:02:57 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:02:57.646449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45ab092-89ea-4e9d-a676-5d0b593eea82 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:02:57.648054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e45ab092-89ea-4e9d-a676-5d0b593eea82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:57.754489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:02:57.771411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45ab092-89ea-4e9d-a676-5d0b593eea82 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:02:57.847059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fbbd95-8ce1-4e98-b46b-ea432cc6d1d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:02:57.847685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbbd95-8ce1-4e98-b46b-ea432cc6d1d5 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:02:57.847963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbbd95-8ce1-4e98-b46b-ea432cc6d1d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:02:57.848190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbbd95-8ce1-4e98-b46b-ea432cc6d1d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:02:57.849850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-56fbbd95-8ce1-4e98-b46b-ea432cc6d1d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:02:57.961804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fbbd95-8ce1-4e98-b46b-ea432cc6d1d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:02:57.963648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fbbd95-8ce1-4e98-b46b-ea432cc6d1d5 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:02:57.982373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 678/1346] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:02:57 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:02.001957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef81ec1-4bdd-4fac-8aab-8574bad22608 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:02.002403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef81ec1-4bdd-4fac-8aab-8574bad22608 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:02.002491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef81ec1-4bdd-4fac-8aab-8574bad22608 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:02.003927 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 669/1347] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:02.010055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6f6a3-f9c4-4ecd-a427-10996bb90f2a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:02.010995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6f6a3-f9c4-4ecd-a427-10996bb90f2a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:02.011553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6f6a3-f9c4-4ecd-a427-10996bb90f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:02.041572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:02.099025 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-23c6f6a3-f9c4-4ecd-a427-10996bb90f2a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:02.100292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-23c6f6a3-f9c4-4ecd-a427-10996bb90f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:02.212838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 679/1348] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:02.251796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef81ec1-4bdd-4fac-8aab-8574bad22608 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:02.253546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1ef81ec1-4bdd-4fac-8aab-8574bad22608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:02.324839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:02.380282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef81ec1-4bdd-4fac-8aab-8574bad22608 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:02.474837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcf594e-9068-43bb-90f5-627927cb9a53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:02.475627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf594e-9068-43bb-90f5-627927cb9a53 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:02.475987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf594e-9068-43bb-90f5-627927cb9a53 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:02.476323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcf594e-9068-43bb-90f5-627927cb9a53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:02.478137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4fcf594e-9068-43bb-90f5-627927cb9a53 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:02.621397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:02.626496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcf594e-9068-43bb-90f5-627927cb9a53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:02.628309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcf594e-9068-43bb-90f5-627927cb9a53 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:02.654921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 670/1349] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:07.551146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b154f11-1806-475b-a575-19b9f2cfc2e9 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:07.551671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b154f11-1806-475b-a575-19b9f2cfc2e9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:07.551950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b154f11-1806-475b-a575-19b9f2cfc2e9 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:07.553422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 680/1350] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:03:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:07.560789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb64da0b-7a02-4514-a327-243fcc343a11 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:07.561263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb64da0b-7a02-4514-a327-243fcc343a11 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:07.561263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb64da0b-7a02-4514-a327-243fcc343a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:07.710985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-bb64da0b-7a02-4514-a327-243fcc343a11 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:07.710985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bb64da0b-7a02-4514-a327-243fcc343a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:07.848340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 671/1351] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:03:07 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:12.065072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257146e5-eb06-4d55-81f0-81b4de28c491 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:12.065995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257146e5-eb06-4d55-81f0-81b4de28c491 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:12.066432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257146e5-eb06-4d55-81f0-81b4de28c491 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:12.068148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 681/1352] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:03:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:12.073482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3ffc70-7994-4c85-a442-596d47c55022 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:12.073990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3ffc70-7994-4c85-a442-596d47c55022 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:12.074072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3ffc70-7994-4c85-a442-596d47c55022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:12.129349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:12.152577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-bb3ffc70-7994-4c85-a442-596d47c55022 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:12.153790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bb3ffc70-7994-4c85-a442-596d47c55022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:12.210716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=59 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:12.268982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 672/1353] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:03:12 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:13.527517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bd411a-5b8c-4371-b517-50305289a718 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:13.528054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bd411a-5b8c-4371-b517-50305289a718 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:13.528136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bd411a-5b8c-4371-b517-50305289a718 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:13.529598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 682/1354] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:03:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:13.534868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8342ba70-2775-4061-9efa-6c814183ddb6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:13.535436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8342ba70-2775-4061-9efa-6c814183ddb6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:13.535436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8342ba70-2775-4061-9efa-6c814183ddb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:13.615230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8342ba70-2775-4061-9efa-6c814183ddb6 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:13.616558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8342ba70-2775-4061-9efa-6c814183ddb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:13.664993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:13.733513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 673/1355] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:03:13 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:03:17.254478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e4952-b48a-453b-aaa1-7e630fc4725f None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:17.255026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e4952-b48a-453b-aaa1-7e630fc4725f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:17.255026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5e4952-b48a-453b-aaa1-7e630fc4725f None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:17.256382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 683/1356] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:17.270472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39003329-aa0d-47a0-a136-da67ee7d9f46 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:17.270472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39003329-aa0d-47a0-a136-da67ee7d9f46 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:17.271195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39003329-aa0d-47a0-a136-da67ee7d9f46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:17.357402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-39003329-aa0d-47a0-a136-da67ee7d9f46 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:17.359051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-39003329-aa0d-47a0-a136-da67ee7d9f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:17.471405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 674/1357] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:17.528667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332593fd-ba1f-4604-8e02-1bd54c53227f None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:17.529275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332593fd-ba1f-4604-8e02-1bd54c53227f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:17.529890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332593fd-ba1f-4604-8e02-1bd54c53227f None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:17.531552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 684/1358] 10.222.0.42 () {54 vars in 962 bytes} [Tue Jul 26 03:03:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:03:17.540938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbc0fa8-d13c-4d38-aa55-712cbdde8708 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:17.541699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbc0fa8-d13c-4d38-aa55-712cbdde8708 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:17.543032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbc0fa8-d13c-4d38-aa55-712cbdde8708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:17.567941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:17.634196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1bbc0fa8-d13c-4d38-aa55-712cbdde8708 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:17.635509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1bbc0fa8-d13c-4d38-aa55-712cbdde8708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:17.775454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 675/1359] 10.222.0.42 () {58 vars in 1059 bytes} [Tue Jul 26 03:03:17 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:17.783264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332593fd-ba1f-4604-8e02-1bd54c53227f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:17.784790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-332593fd-ba1f-4604-8e02-1bd54c53227f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:17.957966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332593fd-ba1f-4604-8e02-1bd54c53227f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:18.044117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:18.089544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c688b2-2e08-4823-ba9f-a29762a36a28 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:18.090634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c688b2-2e08-4823-ba9f-a29762a36a28 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:18.090925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c688b2-2e08-4823-ba9f-a29762a36a28 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:18.091226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c688b2-2e08-4823-ba9f-a29762a36a28 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:18.092967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c0c688b2-2e08-4823-ba9f-a29762a36a28 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:18.187604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c688b2-2e08-4823-ba9f-a29762a36a28 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:18.189203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c688b2-2e08-4823-ba9f-a29762a36a28 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:18.208787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 685/1360] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:03:20.403934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0b562-d7ff-41f4-80e1-1ee3ed04062c None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:20.404440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0b562-d7ff-41f4-80e1-1ee3ed04062c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:20.404641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0b562-d7ff-41f4-80e1-1ee3ed04062c None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:20.406124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 676/1361] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:20.411333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028e1674-26aa-4863-b423-578d1b2367e1 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:20.411760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028e1674-26aa-4863-b423-578d1b2367e1 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:20.411856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028e1674-26aa-4863-b423-578d1b2367e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:20.493931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-028e1674-26aa-4863-b423-578d1b2367e1 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:20.495147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-028e1674-26aa-4863-b423-578d1b2367e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:20.534335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:20.602443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 686/1362] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:03:20.647289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c0b562-d7ff-41f4-80e1-1ee3ed04062c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:20.648735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a9c0b562-d7ff-41f4-80e1-1ee3ed04062c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:20.747167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:20.851837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c0b562-d7ff-41f4-80e1-1ee3ed04062c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:21.015130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c5827c-f158-4505-bb36-e0d57e7eb2f6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:21.015945 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5827c-f158-4505-bb36-e0d57e7eb2f6 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:21.016221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5827c-f158-4505-bb36-e0d57e7eb2f6 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:21.016492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c5827c-f158-4505-bb36-e0d57e7eb2f6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:21.018179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-89c5827c-f158-4505-bb36-e0d57e7eb2f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:21.046756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:21.144517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c5827c-f158-4505-bb36-e0d57e7eb2f6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:21.146368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c5827c-f158-4505-bb36-e0d57e7eb2f6 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:21.167068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 677/1363] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 520 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:23.520185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b650579e-5beb-43d1-b713-33ef99441203 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:23.521078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b650579e-5beb-43d1-b713-33ef99441203 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:23.521361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b650579e-5beb-43d1-b713-33ef99441203 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:23.526590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 687/1364] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:23.528518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a9790-bc1b-4be4-b891-23264ecd5523 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:23.528953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a9790-bc1b-4be4-b891-23264ecd5523 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:23.529306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1a9790-bc1b-4be4-b891-23264ecd5523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:23.614992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7e1a9790-bc1b-4be4-b891-23264ecd5523 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:23.616213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7e1a9790-bc1b-4be4-b891-23264ecd5523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:23.730417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:23.737856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 678/1365] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:23.781120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cda414b-c85b-4829-868d-13242d9731a1 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:23.781942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cda414b-c85b-4829-868d-13242d9731a1 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:23.782377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cda414b-c85b-4829-868d-13242d9731a1 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:23.783915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 688/1366] 10.222.0.42 () {54 vars in 962 bytes} [Tue Jul 26 03:03:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:23.789343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ccd677-046c-441d-9d96-de98dcbcd32d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:23.789791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ccd677-046c-441d-9d96-de98dcbcd32d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:23.790214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ccd677-046c-441d-9d96-de98dcbcd32d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:23.871070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-e0ccd677-046c-441d-9d96-de98dcbcd32d None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:23.871451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e0ccd677-046c-441d-9d96-de98dcbcd32d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:23.997277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 679/1367] 10.222.0.42 () {58 vars in 1059 bytes} [Tue Jul 26 03:03:23 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:24.004591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cda414b-c85b-4829-868d-13242d9731a1 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:24.006197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9cda414b-c85b-4829-868d-13242d9731a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:24.164488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cda414b-c85b-4829-868d-13242d9731a1 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:24.185250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:24.285343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ffd40c-7192-4e34-92e9-85dd3d57b723 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:24.285968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ffd40c-7192-4e34-92e9-85dd3d57b723 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:24.286249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ffd40c-7192-4e34-92e9-85dd3d57b723 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:24.286526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ffd40c-7192-4e34-92e9-85dd3d57b723 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:24.288182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-64ffd40c-7192-4e34-92e9-85dd3d57b723 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:24.382820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ffd40c-7192-4e34-92e9-85dd3d57b723 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:24.384494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ffd40c-7192-4e34-92e9-85dd3d57b723 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:24.404443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 689/1368] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:25.395495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2884f9cc-cab0-4eeb-86cf-3f88713817ce None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:25.396037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2884f9cc-cab0-4eeb-86cf-3f88713817ce None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:25.396037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2884f9cc-cab0-4eeb-86cf-3f88713817ce None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:25.397494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 680/1369] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:25.403029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c2f0a3-d3a6-453d-9d80-81188c934d91 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:25.403498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c2f0a3-d3a6-453d-9d80-81188c934d91 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:25.403779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c2f0a3-d3a6-453d-9d80-81188c934d91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:25.436932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:25.477437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-96c2f0a3-d3a6-453d-9d80-81188c934d91 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:25.478809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-96c2f0a3-d3a6-453d-9d80-81188c934d91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:25.607522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 690/1370] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:25.911900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e19c027-76f9-43f4-9430-1da7f0045178 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:25.912535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e19c027-76f9-43f4-9430-1da7f0045178 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:25.913019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e19c027-76f9-43f4-9430-1da7f0045178 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:25.914592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 681/1371] 10.222.0.42 () {54 vars in 964 bytes} [Tue Jul 26 03:03:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:25.920459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786193cb-0138-48f8-869b-0be529f5f928 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:25.920818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786193cb-0138-48f8-869b-0be529f5f928 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:25.921248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786193cb-0138-48f8-869b-0be529f5f928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:26.012148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-786193cb-0138-48f8-869b-0be529f5f928 None None] MFA Rules not processed for user `48f06347846c48c2933cee7ebd32b947`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:26.012148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-786193cb-0138-48f8-869b-0be529f5f928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:26.062675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:26.096215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 691/1372] 10.222.0.42 () {58 vars in 1061 bytes} [Tue Jul 26 03:03:25 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:26.103751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e19c027-76f9-43f4-9430-1da7f0045178 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:26.105289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4e19c027-76f9-43f4-9430-1da7f0045178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:26.159712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:26.223824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e19c027-76f9-43f4-9430-1da7f0045178 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:26.325982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:26.327238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:26.327526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:26.327809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:26.329573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:26.449961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:26.471316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:26.473100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:26.481051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f976c8-2891-446f-8229-c1091a9cdb0a None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:26.481533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f976c8-2891-446f-8229-c1091a9cdb0a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:26.481636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f976c8-2891-446f-8229-c1091a9cdb0a None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:26.483195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 692/1373] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:03:26.488101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3aefc7-2272-43d1-ba66-992e44f6fc2c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:26.488385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3aefc7-2272-43d1-ba66-992e44f6fc2c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:26.488662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3aefc7-2272-43d1-ba66-992e44f6fc2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:26.494045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 682/1374] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:26.577257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-2b3aefc7-2272-43d1-ba66-992e44f6fc2c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:26.578773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2b3aefc7-2272-43d1-ba66-992e44f6fc2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:26.697466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 693/1375] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:03:26.741934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:26.743190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:26.866805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54b51fc-3b37-41b4-90f2-610c671a9eb2 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:26.971887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:26.984811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c455de-e8f5-412a-84af-eb64a3ef1fd1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:26.986238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c455de-e8f5-412a-84af-eb64a3ef1fd1 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:26.986238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c455de-e8f5-412a-84af-eb64a3ef1fd1 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:26.986238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c455de-e8f5-412a-84af-eb64a3ef1fd1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:26.987919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b0c455de-e8f5-412a-84af-eb64a3ef1fd1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:27.079923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c455de-e8f5-412a-84af-eb64a3ef1fd1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:27.081549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c455de-e8f5-412a-84af-eb64a3ef1fd1 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:27.100043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 683/1376] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:29.153752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0bfa00-a438-4d0b-a0b0-9999488c7577 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:29.154344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0bfa00-a438-4d0b-a0b0-9999488c7577 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:29.154344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0bfa00-a438-4d0b-a0b0-9999488c7577 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:29.155889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 694/1377] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:29.161509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376e1575-27c6-4e8d-8e46-d2419288b45b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:29.161997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376e1575-27c6-4e8d-8e46-d2419288b45b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:29.162174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376e1575-27c6-4e8d-8e46-d2419288b45b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:29.231916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-376e1575-27c6-4e8d-8e46-d2419288b45b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:29.233488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-376e1575-27c6-4e8d-8e46-d2419288b45b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:29.295655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:29.339346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 684/1378] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:03:29.381657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0bfa00-a438-4d0b-a0b0-9999488c7577 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:29.383311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6e0bfa00-a438-4d0b-a0b0-9999488c7577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:29.439125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:29.550350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0bfa00-a438-4d0b-a0b0-9999488c7577 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:29.673781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2482418b-f071-40e7-ae56-dcaac404ab14 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:29.674405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2482418b-f071-40e7-ae56-dcaac404ab14 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:29.674680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2482418b-f071-40e7-ae56-dcaac404ab14 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:29.674962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2482418b-f071-40e7-ae56-dcaac404ab14 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:29.676694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2482418b-f071-40e7-ae56-dcaac404ab14 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:29.689263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:29.772638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2482418b-f071-40e7-ae56-dcaac404ab14 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:29.774320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2482418b-f071-40e7-ae56-dcaac404ab14 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:29.794224 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 695/1379] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:31.842299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7150ece3-a2b0-4eb1-aa22-cbf2c1d5738b None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:31.842871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7150ece3-a2b0-4eb1-aa22-cbf2c1d5738b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:31.842871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7150ece3-a2b0-4eb1-aa22-cbf2c1d5738b None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:31.844495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 685/1380] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:31.849857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90530b7-d36f-4748-8b55-4ae413f91ae7 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:31.850349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90530b7-d36f-4748-8b55-4ae413f91ae7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:31.850568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90530b7-d36f-4748-8b55-4ae413f91ae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:31.925401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e90530b7-d36f-4748-8b55-4ae413f91ae7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:31.926655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e90530b7-d36f-4748-8b55-4ae413f91ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:32.028409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:32.043721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 696/1381] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:32.090206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7150ece3-a2b0-4eb1-aa22-cbf2c1d5738b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:32.092099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7150ece3-a2b0-4eb1-aa22-cbf2c1d5738b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:32.251091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7150ece3-a2b0-4eb1-aa22-cbf2c1d5738b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:32.303333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:32.381994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebca074f-c42d-4f07-9c1b-d0f185ee2760 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:32.382982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebca074f-c42d-4f07-9c1b-d0f185ee2760 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:32.383463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebca074f-c42d-4f07-9c1b-d0f185ee2760 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:32.383921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebca074f-c42d-4f07-9c1b-d0f185ee2760 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:32.385844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ebca074f-c42d-4f07-9c1b-d0f185ee2760 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:32.484279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebca074f-c42d-4f07-9c1b-d0f185ee2760 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:32.485933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebca074f-c42d-4f07-9c1b-d0f185ee2760 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:32.504950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 686/1382] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:34.574148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9afd7ee-62c5-4849-b2a0-1327285c36bc None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:34.574759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9afd7ee-62c5-4849-b2a0-1327285c36bc None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:34.575265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9afd7ee-62c5-4849-b2a0-1327285c36bc None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:34.577398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 697/1383] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:34.583195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8411c954-7bec-4c1b-9e16-452351729c5b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:34.583662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8411c954-7bec-4c1b-9e16-452351729c5b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:34.583783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8411c954-7bec-4c1b-9e16-452351729c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:34.663454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8411c954-7bec-4c1b-9e16-452351729c5b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:34.664683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8411c954-7bec-4c1b-9e16-452351729c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:34.666929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:34.773874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 687/1384] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:34.819822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9afd7ee-62c5-4849-b2a0-1327285c36bc None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:34.822345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d9afd7ee-62c5-4849-b2a0-1327285c36bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:35.003547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9afd7ee-62c5-4849-b2a0-1327285c36bc None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:35.070105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:35.129421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233a9563-cd10-4321-beda-5f8ffa894363 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:35.130089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a9563-cd10-4321-beda-5f8ffa894363 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:35.130385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a9563-cd10-4321-beda-5f8ffa894363 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:35.130668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233a9563-cd10-4321-beda-5f8ffa894363 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:35.132466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-233a9563-cd10-4321-beda-5f8ffa894363 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:35.229414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a9563-cd10-4321-beda-5f8ffa894363 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:35.231436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233a9563-cd10-4321-beda-5f8ffa894363 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:35.251849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 698/1385] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:37.626097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70816bcf-1311-4f93-a32f-a27e8bff1288 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:37.626687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70816bcf-1311-4f93-a32f-a27e8bff1288 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:37.626687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70816bcf-1311-4f93-a32f-a27e8bff1288 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:37.628266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 688/1386] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:37.634698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456862d-e2b3-4134-8791-637b7d0523a6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:37.635199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456862d-e2b3-4134-8791-637b7d0523a6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:37.635311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456862d-e2b3-4134-8791-637b7d0523a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:37.720761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d456862d-e2b3-4134-8791-637b7d0523a6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:37.722601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d456862d-e2b3-4134-8791-637b7d0523a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:37.759278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:37.847950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 699/1387] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:37.894515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70816bcf-1311-4f93-a32f-a27e8bff1288 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:37.896231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-70816bcf-1311-4f93-a32f-a27e8bff1288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:38.046019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:38.066265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70816bcf-1311-4f93-a32f-a27e8bff1288 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:38.202639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b892be93-29ba-48f4-b6be-2c27f00bc429 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:38.203278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b892be93-29ba-48f4-b6be-2c27f00bc429 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:38.203566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b892be93-29ba-48f4-b6be-2c27f00bc429 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:38.203849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b892be93-29ba-48f4-b6be-2c27f00bc429 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:38.205635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b892be93-29ba-48f4-b6be-2c27f00bc429 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:38.318595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b892be93-29ba-48f4-b6be-2c27f00bc429 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:38.320394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b892be93-29ba-48f4-b6be-2c27f00bc429 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:38.325874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:38.343582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 689/1388] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:40.443973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430d9507-fca1-4f42-80a5-e49ecccbee86 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:40.444557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430d9507-fca1-4f42-80a5-e49ecccbee86 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:40.444557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430d9507-fca1-4f42-80a5-e49ecccbee86 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:40.446184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 700/1389] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:40.451151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d882dae-6cf3-4584-a079-e9671315fd79 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:40.451635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d882dae-6cf3-4584-a079-e9671315fd79 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:40.451635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d882dae-6cf3-4584-a079-e9671315fd79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:40.519791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7d882dae-6cf3-4584-a079-e9671315fd79 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:40.520991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7d882dae-6cf3-4584-a079-e9671315fd79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:40.628761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 690/1390] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:40.674706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430d9507-fca1-4f42-80a5-e49ecccbee86 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:40.676305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-430d9507-fca1-4f42-80a5-e49ecccbee86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:40.825269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430d9507-fca1-4f42-80a5-e49ecccbee86 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:40.834560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:40.941520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa73eb4b-cc48-48e8-821c-691a4f6a9a31 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:40.942053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73eb4b-cc48-48e8-821c-691a4f6a9a31 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:40.942340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73eb4b-cc48-48e8-821c-691a4f6a9a31 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:40.942636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73eb4b-cc48-48e8-821c-691a4f6a9a31 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:40.944299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-aa73eb4b-cc48-48e8-821c-691a4f6a9a31 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:41.037458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa73eb4b-cc48-48e8-821c-691a4f6a9a31 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:41.039152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa73eb4b-cc48-48e8-821c-691a4f6a9a31 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:41.058107 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 701/1391] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:03:41.554483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7311f724-739d-4432-b282-cdb98c627ab1 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:41.555339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7311f724-739d-4432-b282-cdb98c627ab1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:41.555339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7311f724-739d-4432-b282-cdb98c627ab1 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:41.556517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 691/1392] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:41.563529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e8ac58-9c64-4af1-9f7f-dc44818aa4d7 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:41.563529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e8ac58-9c64-4af1-9f7f-dc44818aa4d7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:41.564178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e8ac58-9c64-4af1-9f7f-dc44818aa4d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:41.579823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:41.635419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-43e8ac58-9c64-4af1-9f7f-dc44818aa4d7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:41.636646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-43e8ac58-9c64-4af1-9f7f-dc44818aa4d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:41.746603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 702/1393] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:41.803980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baa572f-3284-4960-95b5-6e867f000f0e None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:41.804346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baa572f-3284-4960-95b5-6e867f000f0e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:41.804869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baa572f-3284-4960-95b5-6e867f000f0e None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:41.806249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 692/1394] 10.222.0.42 () {54 vars in 966 bytes} [Tue Jul 26 03:03:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:41.811704 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80b7d31-29ae-441b-ae16-e782b8ededce None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:41.811996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80b7d31-29ae-441b-ae16-e782b8ededce None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:41.812267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80b7d31-29ae-441b-ae16-e782b8ededce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:41.893363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a80b7d31-29ae-441b-ae16-e782b8ededce None None] MFA Rules not processed for user `81ec7fc0b48643739402efba1285c713`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:41.894587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a80b7d31-29ae-441b-ae16-e782b8ededce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:41.930068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:41.973101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 703/1395] 10.222.0.42 () {58 vars in 1063 bytes} [Tue Jul 26 03:03:41 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:41.980969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baa572f-3284-4960-95b5-6e867f000f0e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:41.982541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9baa572f-3284-4960-95b5-6e867f000f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:42.067385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:42.101890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baa572f-3284-4960-95b5-6e867f000f0e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:42.181234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c63882-7f55-4997-b6d1-eb593a2338a0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:42.181868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c63882-7f55-4997-b6d1-eb593a2338a0 service glance] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:42.182149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c63882-7f55-4997-b6d1-eb593a2338a0 service glance] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:42.182428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c63882-7f55-4997-b6d1-eb593a2338a0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:42.184092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-57c63882-7f55-4997-b6d1-eb593a2338a0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:42.298295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c63882-7f55-4997-b6d1-eb593a2338a0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:42.299764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c63882-7f55-4997-b6d1-eb593a2338a0 service glance] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:42.315920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:42.321500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 693/1396] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:43.062325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06e92b-bd93-4da5-9880-d3b9e0bd8725 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:43.062836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06e92b-bd93-4da5-9880-d3b9e0bd8725 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:43.062969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06e92b-bd93-4da5-9880-d3b9e0bd8725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:43.130462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4b06e92b-bd93-4da5-9880-d3b9e0bd8725 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:43.131708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4b06e92b-bd93-4da5-9880-d3b9e0bd8725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:43.164248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd3e06f-1676-47b1-be6c-80d82814f918 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:43.164834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd3e06f-1676-47b1-be6c-80d82814f918 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:43.164834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd3e06f-1676-47b1-be6c-80d82814f918 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:43.166153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 694/1397] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:43.171348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:43.171999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:43.171999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:43.221779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 704/1398] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:03:43 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:43.241389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:43.242609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:43.269447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b0c2bc-e6a7-47a9-bf8d-fedffce3381e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:43.269649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b0c2bc-e6a7-47a9-bf8d-fedffce3381e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:43.269910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b0c2bc-e6a7-47a9-bf8d-fedffce3381e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:43.271889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 705/1399] 10.222.0.42 () {54 vars in 943 bytes} [Tue Jul 26 03:03:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:43.277325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c1a739-4079-4614-baf0-72ec5c1d61d9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:43.277608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c1a739-4079-4614-baf0-72ec5c1d61d9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:43.277881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c1a739-4079-4614-baf0-72ec5c1d61d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:43.352975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:43.353394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 695/1400] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:03:43.374208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-58c1a739-4079-4614-baf0-72ec5c1d61d9 None None] MFA Rules not processed for user `afebddca912546119eea61f568bbd60d`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:43.375441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-58c1a739-4079-4614-baf0-72ec5c1d61d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:43.396999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:43.398335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:43.477937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:43.484672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 706/1401] 10.222.0.42 () {58 vars in 1040 bytes} [Tue Jul 26 03:03:43 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:43.493939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c1a739-4079-4614-baf0-72ec5c1d61d9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:43.503554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c1a739-4079-4614-baf0-72ec5c1d61d9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:43.516727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fac4db9-e4b7-4ef7-bfe4-250dfe61d8c9 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:43.605874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:43.611948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b38291-e434-47d7-aec9-817aa6f8256d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:43.612557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b38291-e434-47d7-aec9-817aa6f8256d service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:43.612834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b38291-e434-47d7-aec9-817aa6f8256d service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:43.613105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b38291-e434-47d7-aec9-817aa6f8256d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:43.614803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-23b38291-e434-47d7-aec9-817aa6f8256d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:43.636083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2d9378-dd8e-459b-a977-0e0a2d7d79bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:43.636712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d9378-dd8e-459b-a977-0e0a2d7d79bc service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:43.636993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d9378-dd8e-459b-a977-0e0a2d7d79bc service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:43.637259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2d9378-dd8e-459b-a977-0e0a2d7d79bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:43.639015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6d2d9378-dd8e-459b-a977-0e0a2d7d79bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:43.702305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b38291-e434-47d7-aec9-817aa6f8256d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:43.704121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b38291-e434-47d7-aec9-817aa6f8256d service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:43.712236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 707/1402] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:03:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 26 03:03:43.718724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:43.737130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2d9378-dd8e-459b-a977-0e0a2d7d79bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:43.738754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2d9378-dd8e-459b-a977-0e0a2d7d79bc service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:43.759765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 696/1403] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:03:45.819215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f50ff0-2a7c-474b-be0f-745567c35f19 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:45.819849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f50ff0-2a7c-474b-be0f-745567c35f19 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:45.819849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f50ff0-2a7c-474b-be0f-745567c35f19 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:45.821122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 708/1404] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:45.827253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6112f5-fdbf-4714-bea5-8449a3949187 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:45.827750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6112f5-fdbf-4714-bea5-8449a3949187 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:45.828020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6112f5-fdbf-4714-bea5-8449a3949187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:45.906157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1c6112f5-fdbf-4714-bea5-8449a3949187 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:45.907925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1c6112f5-fdbf-4714-bea5-8449a3949187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:46.016267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 697/1405] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:46.062829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f50ff0-2a7c-474b-be0f-745567c35f19 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:46.064405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-36f50ff0-2a7c-474b-be0f-745567c35f19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:46.224668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:46.239957 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f50ff0-2a7c-474b-be0f-745567c35f19 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:46.364384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b125d330-c48c-4195-b201-f0f5fd9646df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:46.365009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125d330-c48c-4195-b201-f0f5fd9646df service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:46.365283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125d330-c48c-4195-b201-f0f5fd9646df service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:46.365553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125d330-c48c-4195-b201-f0f5fd9646df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:46.367222 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b125d330-c48c-4195-b201-f0f5fd9646df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:46.474908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b125d330-c48c-4195-b201-f0f5fd9646df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:46.476597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b125d330-c48c-4195-b201-f0f5fd9646df service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:46.487967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:46.496762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 709/1406] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:48.519499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdf884e-c385-4c56-8f4d-91a5aaee4d6f None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:48.520100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdf884e-c385-4c56-8f4d-91a5aaee4d6f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:48.520100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdf884e-c385-4c56-8f4d-91a5aaee4d6f None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:48.521407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 698/1407] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:03:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:03:48.526678 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63710662-6a67-40c6-955c-f7790f62155e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:48.527158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63710662-6a67-40c6-955c-f7790f62155e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:48.527262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63710662-6a67-40c6-955c-f7790f62155e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:48.599283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-63710662-6a67-40c6-955c-f7790f62155e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:03:48.600481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-63710662-6a67-40c6-955c-f7790f62155e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:48.719995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 710/1408] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:03:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:03:48.764114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdf884e-c385-4c56-8f4d-91a5aaee4d6f None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:03:48.765969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6cdf884e-c385-4c56-8f4d-91a5aaee4d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:48.830095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:48.939129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdf884e-c385-4c56-8f4d-91a5aaee4d6f None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:03:49.065943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd91adbc-2d12-46b2-a5c1-5b2aa7ff69ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:03:49.066595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd91adbc-2d12-46b2-a5c1-5b2aa7ff69ae service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:03:49.067018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd91adbc-2d12-46b2-a5c1-5b2aa7ff69ae service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:03:49.067298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd91adbc-2d12-46b2-a5c1-5b2aa7ff69ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:03:49.069067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bd91adbc-2d12-46b2-a5c1-5b2aa7ff69ae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:03:49.086071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:49.176324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd91adbc-2d12-46b2-a5c1-5b2aa7ff69ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:03:49.178042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd91adbc-2d12-46b2-a5c1-5b2aa7ff69ae service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:03:49.198620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 699/1409] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:03:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:03:58.717912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=67 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:03:59.194148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=39 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:05.904389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53c4c4-660b-477e-871b-5597aa742ca7 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:05.905024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53c4c4-660b-477e-871b-5597aa742ca7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:05.905024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53c4c4-660b-477e-871b-5597aa742ca7 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:05.906312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 711/1410] 10.222.0.42 () {54 vars in 918 bytes} [Tue Jul 26 03:04:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:05.911768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275b54c4-26d7-4842-8087-2ad7730d9330 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:05.912229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275b54c4-26d7-4842-8087-2ad7730d9330 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:05.912331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275b54c4-26d7-4842-8087-2ad7730d9330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:06.002581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-275b54c4-26d7-4842-8087-2ad7730d9330 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:06.003843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-275b54c4-26d7-4842-8087-2ad7730d9330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:06.131562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 700/1411] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Jul 26 03:04:05 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:06.146570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8376d1-cba3-4412-9867-f8ad9e9794e8 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:06.146856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8376d1-cba3-4412-9867-f8ad9e9794e8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:06.147181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8376d1-cba3-4412-9867-f8ad9e9794e8 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:06.148574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 712/1412] 10.222.0.42 () {54 vars in 918 bytes} [Tue Jul 26 03:04:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:06.155994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be0c99c-2e7c-41f9-b2cf-2e17dc0ee70d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:06.156550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be0c99c-2e7c-41f9-b2cf-2e17dc0ee70d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:06.156656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be0c99c-2e7c-41f9-b2cf-2e17dc0ee70d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:06.249089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:06.252532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4be0c99c-2e7c-41f9-b2cf-2e17dc0ee70d None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:06.254455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4be0c99c-2e7c-41f9-b2cf-2e17dc0ee70d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:06.387687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 701/1413] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Jul 26 03:04:06 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:08.911560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7297e-ca62-4177-9953-6bd877a4bfca None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:08.912498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7297e-ca62-4177-9953-6bd877a4bfca None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:08.912498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb7297e-ca62-4177-9953-6bd877a4bfca None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:08.914606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 713/1414] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:08.921739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4cc33-d5ff-4c14-88f3-2a633ba19f19 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:08.922197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4cc33-d5ff-4c14-88f3-2a633ba19f19 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:08.922297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4cc33-d5ff-4c14-88f3-2a633ba19f19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:08.997388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-0aa4cc33-d5ff-4c14-88f3-2a633ba19f19 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:08.999217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0aa4cc33-d5ff-4c14-88f3-2a633ba19f19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:09.061045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:09.110867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 702/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:09.157497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb7297e-ca62-4177-9953-6bd877a4bfca None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:09.160034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dfb7297e-ca62-4177-9953-6bd877a4bfca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:09.360639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb7297e-ca62-4177-9953-6bd877a4bfca None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:09.439593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:09.480876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e375f72f-88e2-449a-9bd6-66ced6dc6e68 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:09.481572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e375f72f-88e2-449a-9bd6-66ced6dc6e68 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:09.481881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e375f72f-88e2-449a-9bd6-66ced6dc6e68 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:09.482154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e375f72f-88e2-449a-9bd6-66ced6dc6e68 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:09.483866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e375f72f-88e2-449a-9bd6-66ced6dc6e68 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:09.580908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e375f72f-88e2-449a-9bd6-66ced6dc6e68 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:09.582531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e375f72f-88e2-449a-9bd6-66ced6dc6e68 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:09.605410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 714/1416] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:10.518383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc9803-04b2-4fe7-94b5-9e1990381cc3 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:10.519309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc9803-04b2-4fe7-94b5-9e1990381cc3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:10.519309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc9803-04b2-4fe7-94b5-9e1990381cc3 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:10.521557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 703/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:10.527547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e083e3a9-39ba-4892-ad0b-a52fbd7f9fd4 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:10.528351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e083e3a9-39ba-4892-ad0b-a52fbd7f9fd4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:10.528541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e083e3a9-39ba-4892-ad0b-a52fbd7f9fd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:10.610478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e083e3a9-39ba-4892-ad0b-a52fbd7f9fd4 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:10.611690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e083e3a9-39ba-4892-ad0b-a52fbd7f9fd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:10.642866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:10.727322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 715/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:10.881869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc9803-04b2-4fe7-94b5-9e1990381cc3 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:10.884205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9bfc9803-04b2-4fe7-94b5-9e1990381cc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:11.047771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc9803-04b2-4fe7-94b5-9e1990381cc3 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:11.092270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:11.166277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fbe68d-7bff-46e9-b01a-ce766f4165ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:11.166903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbe68d-7bff-46e9-b01a-ce766f4165ee service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:11.167277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbe68d-7bff-46e9-b01a-ce766f4165ee service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:11.167550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbe68d-7bff-46e9-b01a-ce766f4165ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:11.169215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b0fbe68d-7bff-46e9-b01a-ce766f4165ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:11.260882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fbe68d-7bff-46e9-b01a-ce766f4165ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:11.262486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fbe68d-7bff-46e9-b01a-ce766f4165ee service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:11.281227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 704/1419] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:12.171457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68ea15-2162-470d-8406-7a5c3b7af260 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:12.172026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68ea15-2162-470d-8406-7a5c3b7af260 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:12.172026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68ea15-2162-470d-8406-7a5c3b7af260 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:12.173391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 716/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:12.181890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d4a50-adc2-4a42-ae21-cd6c4802dfe2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:12.182678 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d4a50-adc2-4a42-ae21-cd6c4802dfe2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:12.182812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56d4a50-adc2-4a42-ae21-cd6c4802dfe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:12.249745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:12.272959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-e56d4a50-adc2-4a42-ae21-cd6c4802dfe2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:12.274142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e56d4a50-adc2-4a42-ae21-cd6c4802dfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:12.386989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 705/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:12.545014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e68ea15-2162-470d-8406-7a5c3b7af260 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:12.547757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3e68ea15-2162-470d-8406-7a5c3b7af260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:12.705973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e68ea15-2162-470d-8406-7a5c3b7af260 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:12.719162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:12.825594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407fc939-9ab9-4c00-8c7a-4cd383ed8b10 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:12.826132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407fc939-9ab9-4c00-8c7a-4cd383ed8b10 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:12.826407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407fc939-9ab9-4c00-8c7a-4cd383ed8b10 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:12.826703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407fc939-9ab9-4c00-8c7a-4cd383ed8b10 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:12.828363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-407fc939-9ab9-4c00-8c7a-4cd383ed8b10 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:12.922280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407fc939-9ab9-4c00-8c7a-4cd383ed8b10 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:12.924717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407fc939-9ab9-4c00-8c7a-4cd383ed8b10 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:12.952239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 717/1422] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:13.811239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c93043-6a83-4072-94b8-d8aed30401fb None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:13.811841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c93043-6a83-4072-94b8-d8aed30401fb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:13.811841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c93043-6a83-4072-94b8-d8aed30401fb None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:13.813876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 706/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:13.821486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a82857-7315-4dd4-b1eb-2b4f66c0cc83 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:13.821899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a82857-7315-4dd4-b1eb-2b4f66c0cc83 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:13.822088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a82857-7315-4dd4-b1eb-2b4f66c0cc83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:13.843960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:13.902548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-17a82857-7315-4dd4-b1eb-2b4f66c0cc83 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:13.903803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-17a82857-7315-4dd4-b1eb-2b4f66c0cc83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:14.020558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 718/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:14.180749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c93043-6a83-4072-94b8-d8aed30401fb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:14.183294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-12c93043-6a83-4072-94b8-d8aed30401fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:14.305744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:14.342742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c93043-6a83-4072-94b8-d8aed30401fb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:14.466259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1a8f55-742c-48f9-a638-9708873513fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:14.466790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8f55-742c-48f9-a638-9708873513fc service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:14.467102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8f55-742c-48f9-a638-9708873513fc service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:14.467371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1a8f55-742c-48f9-a638-9708873513fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:14.468987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3e1a8f55-742c-48f9-a638-9708873513fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:14.548289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:14.565162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a8f55-742c-48f9-a638-9708873513fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:14.566778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1a8f55-742c-48f9-a638-9708873513fc service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:14.585612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 707/1425] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:15.447236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4313e-9466-4f6a-8150-07dd5666a4f3 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:15.447809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4313e-9466-4f6a-8150-07dd5666a4f3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:15.447809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4313e-9466-4f6a-8150-07dd5666a4f3 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:15.449154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 719/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:15.455470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507c3ef-f5e3-45ea-b996-bbc780f32a29 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:15.455973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507c3ef-f5e3-45ea-b996-bbc780f32a29 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:15.456065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7507c3ef-f5e3-45ea-b996-bbc780f32a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:15.692178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7507c3ef-f5e3-45ea-b996-bbc780f32a29 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:15.693413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7507c3ef-f5e3-45ea-b996-bbc780f32a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:15.806939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 708/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:15.957949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4313e-9466-4f6a-8150-07dd5666a4f3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:15.960313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0aa4313e-9466-4f6a-8150-07dd5666a4f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:16.058089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:16.123762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4313e-9466-4f6a-8150-07dd5666a4f3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:16.247556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0040801-f35c-4a10-836d-c69e9a0282a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:16.248183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0040801-f35c-4a10-836d-c69e9a0282a6 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:16.248465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0040801-f35c-4a10-836d-c69e9a0282a6 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:16.248736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0040801-f35c-4a10-836d-c69e9a0282a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:16.250357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d0040801-f35c-4a10-836d-c69e9a0282a6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:16.304375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:16.348985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0040801-f35c-4a10-836d-c69e9a0282a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:16.351394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0040801-f35c-4a10-836d-c69e9a0282a6 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:16.377761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 720/1428] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:17.191860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d285fd-fe8b-46b1-95b0-dfcc3fe1e4a5 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:17.192749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d285fd-fe8b-46b1-95b0-dfcc3fe1e4a5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:17.192749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d285fd-fe8b-46b1-95b0-dfcc3fe1e4a5 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:17.194987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 709/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:17.202786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e0c9e2-137b-41fe-b259-5aaa44b3ee39 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:17.203265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e0c9e2-137b-41fe-b259-5aaa44b3ee39 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:17.203376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e0c9e2-137b-41fe-b259-5aaa44b3ee39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:17.283428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f8e0c9e2-137b-41fe-b259-5aaa44b3ee39 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:17.284545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f8e0c9e2-137b-41fe-b259-5aaa44b3ee39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:17.505750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 721/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:04:17.659745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d285fd-fe8b-46b1-95b0-dfcc3fe1e4a5 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:17.661263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-29d285fd-fe8b-46b1-95b0-dfcc3fe1e4a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:17.712023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:17.817292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d285fd-fe8b-46b1-95b0-dfcc3fe1e4a5 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:17.947843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9961586-1712-447a-a2cf-392dc6673d73 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:17.948381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9961586-1712-447a-a2cf-392dc6673d73 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:17.948653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9961586-1712-447a-a2cf-392dc6673d73 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:17.948921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9961586-1712-447a-a2cf-392dc6673d73 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:17.950565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d9961586-1712-447a-a2cf-392dc6673d73 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:17.958241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:18.046347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9961586-1712-447a-a2cf-392dc6673d73 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:18.048027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9961586-1712-447a-a2cf-392dc6673d73 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:18.067178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 710/1431] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:18.872884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb26bf9-555f-4486-9189-4e6aa04ec626 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:18.873577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb26bf9-555f-4486-9189-4e6aa04ec626 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:18.873577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb26bf9-555f-4486-9189-4e6aa04ec626 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:18.874773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 722/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:18.880986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591edb0-0767-450c-ad13-f2c99e0cec04 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:18.881777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591edb0-0767-450c-ad13-f2c99e0cec04 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:18.881925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d591edb0-0767-450c-ad13-f2c99e0cec04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:18.963791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d591edb0-0767-450c-ad13-f2c99e0cec04 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:18.964980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d591edb0-0767-450c-ad13-f2c99e0cec04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:19.034579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:19.064006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 711/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:19.218135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb26bf9-555f-4486-9189-4e6aa04ec626 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:19.220212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9eb26bf9-555f-4486-9189-4e6aa04ec626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:19.242648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:19.374409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb26bf9-555f-4486-9189-4e6aa04ec626 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:19.469589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:19.496440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e45ca3-fb63-4e5f-997b-206d85cdf872 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:19.497134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e45ca3-fb63-4e5f-997b-206d85cdf872 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:19.497406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e45ca3-fb63-4e5f-997b-206d85cdf872 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:19.497672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e45ca3-fb63-4e5f-997b-206d85cdf872 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:19.499339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-09e45ca3-fb63-4e5f-997b-206d85cdf872 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:19.590735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e45ca3-fb63-4e5f-997b-206d85cdf872 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:19.592333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e45ca3-fb63-4e5f-997b-206d85cdf872 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:19.610838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 723/1434] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:04:20.403993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9efa1a-57b7-40d6-920b-90d472491c06 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:20.404587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9efa1a-57b7-40d6-920b-90d472491c06 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:20.404587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9efa1a-57b7-40d6-920b-90d472491c06 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:20.405900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 712/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:20.411969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0b060e-ce88-49b7-ac3f-c41dab80ccdf None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:20.412541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0b060e-ce88-49b7-ac3f-c41dab80ccdf None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:20.412541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0b060e-ce88-49b7-ac3f-c41dab80ccdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:20.505531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5a0b060e-ce88-49b7-ac3f-c41dab80ccdf None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:20.506726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5a0b060e-ce88-49b7-ac3f-c41dab80ccdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:20.548955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:20.606112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 724/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:20.767924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9efa1a-57b7-40d6-920b-90d472491c06 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:20.769356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2a9efa1a-57b7-40d6-920b-90d472491c06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:20.923527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9efa1a-57b7-40d6-920b-90d472491c06 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:20.981111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:21.042252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4866495f-63c9-4035-b795-8567fbcd627b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:21.042815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4866495f-63c9-4035-b795-8567fbcd627b service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:21.043122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4866495f-63c9-4035-b795-8567fbcd627b service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:21.043368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4866495f-63c9-4035-b795-8567fbcd627b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:21.045035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4866495f-63c9-4035-b795-8567fbcd627b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:21.140505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4866495f-63c9-4035-b795-8567fbcd627b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:21.142107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4866495f-63c9-4035-b795-8567fbcd627b service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:21.160429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 713/1437] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:21.963712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d54e83f-7b38-46af-b2d3-a77debcfa4bd None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:21.964232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d54e83f-7b38-46af-b2d3-a77debcfa4bd None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:21.964232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d54e83f-7b38-46af-b2d3-a77debcfa4bd None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:21.965624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 725/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:04:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:21.970758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7034de8-3465-4074-81c5-7d41a1d2014c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:21.971236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7034de8-3465-4074-81c5-7d41a1d2014c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:21.971329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7034de8-3465-4074-81c5-7d41a1d2014c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:22.052402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b7034de8-3465-4074-81c5-7d41a1d2014c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:22.053600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b7034de8-3465-4074-81c5-7d41a1d2014c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:22.055733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:22.155973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 714/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:04:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:22.310129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d54e83f-7b38-46af-b2d3-a77debcfa4bd None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:22.311624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7d54e83f-7b38-46af-b2d3-a77debcfa4bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:22.469954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d54e83f-7b38-46af-b2d3-a77debcfa4bd None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:22.498060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:22.589522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3906cc-2173-47ee-b852-6d9df51d74d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:22.590057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3906cc-2173-47ee-b852-6d9df51d74d3 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:22.590330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3906cc-2173-47ee-b852-6d9df51d74d3 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:22.590641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3906cc-2173-47ee-b852-6d9df51d74d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:22.592293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bd3906cc-2173-47ee-b852-6d9df51d74d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:22.691080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3906cc-2173-47ee-b852-6d9df51d74d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:22.692658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3906cc-2173-47ee-b852-6d9df51d74d3 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:22.710736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 726/1440] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:24.632469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b805fe-c6f3-4147-bbe7-1d9e58e63f40 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:24.633363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b805fe-c6f3-4147-bbe7-1d9e58e63f40 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:24.633363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b805fe-c6f3-4147-bbe7-1d9e58e63f40 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:24.635741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 715/1441] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:24.641298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99f7ad5-b4a6-4b78-9ada-9c903bf802e9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:24.641768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99f7ad5-b4a6-4b78-9ada-9c903bf802e9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:24.641870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99f7ad5-b4a6-4b78-9ada-9c903bf802e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:24.683579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:24.710035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f99f7ad5-b4a6-4b78-9ada-9c903bf802e9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:24.711263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f99f7ad5-b4a6-4b78-9ada-9c903bf802e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:24.811060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 727/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:24.853044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b805fe-c6f3-4147-bbe7-1d9e58e63f40 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:24.854761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f5b805fe-c6f3-4147-bbe7-1d9e58e63f40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:24.978885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b805fe-c6f3-4147-bbe7-1d9e58e63f40 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:25.017975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:25.061676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5992cd-18ee-44fe-bb0d-69ebcc9cad21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:25.062305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5992cd-18ee-44fe-bb0d-69ebcc9cad21 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:25.062583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5992cd-18ee-44fe-bb0d-69ebcc9cad21 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:25.062879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5992cd-18ee-44fe-bb0d-69ebcc9cad21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:25.064565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5b5992cd-18ee-44fe-bb0d-69ebcc9cad21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:25.195009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5992cd-18ee-44fe-bb0d-69ebcc9cad21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:25.196507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5992cd-18ee-44fe-bb0d-69ebcc9cad21 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:25.216002 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 716/1443] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:27.346107 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0f087d-04a0-48d6-93f4-ac321eb7ba8d None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:27.346668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0f087d-04a0-48d6-93f4-ac321eb7ba8d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:27.346668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0f087d-04a0-48d6-93f4-ac321eb7ba8d None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:27.347963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 728/1444] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:27.354558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144f5724-5af8-4625-9a1f-552079324fc9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:27.355022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144f5724-5af8-4625-9a1f-552079324fc9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:27.355143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144f5724-5af8-4625-9a1f-552079324fc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:27.435603 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-144f5724-5af8-4625-9a1f-552079324fc9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:27.436825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-144f5724-5af8-4625-9a1f-552079324fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:27.450348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:27.565911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 717/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:27.608129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0f087d-04a0-48d6-93f4-ac321eb7ba8d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:27.609655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-de0f087d-04a0-48d6-93f4-ac321eb7ba8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:27.717378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0f087d-04a0-48d6-93f4-ac321eb7ba8d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:27.730483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:27.800794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2070d1-8b82-4855-9597-17fa62732e6c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:27.801421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2070d1-8b82-4855-9597-17fa62732e6c service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:27.801695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2070d1-8b82-4855-9597-17fa62732e6c service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:27.801995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2070d1-8b82-4855-9597-17fa62732e6c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:27.803739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ad2070d1-8b82-4855-9597-17fa62732e6c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:27.944006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2070d1-8b82-4855-9597-17fa62732e6c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:27.945451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2070d1-8b82-4855-9597-17fa62732e6c service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:27.964277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 729/1446] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:30.299146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e70e4f-7d31-4655-afbb-99822de8ed53 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:30.299949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e70e4f-7d31-4655-afbb-99822de8ed53 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:30.300181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e70e4f-7d31-4655-afbb-99822de8ed53 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:30.302597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 718/1447] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:30.311498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07d9a7e-c57d-4f26-b939-131724e42d6c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:30.311916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07d9a7e-c57d-4f26-b939-131724e42d6c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:30.312023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07d9a7e-c57d-4f26-b939-131724e42d6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:30.366929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:30.389079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f07d9a7e-c57d-4f26-b939-131724e42d6c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:30.390299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f07d9a7e-c57d-4f26-b939-131724e42d6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:30.489761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 730/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:30.541162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e70e4f-7d31-4655-afbb-99822de8ed53 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:30.543606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b7e70e4f-7d31-4655-afbb-99822de8ed53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:30.663309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e70e4f-7d31-4655-afbb-99822de8ed53 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:30.709338 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:30.745841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae25440-ba5d-42f5-b31a-d3324a4e12aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:30.746461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae25440-ba5d-42f5-b31a-d3324a4e12aa service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:30.746820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae25440-ba5d-42f5-b31a-d3324a4e12aa service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:30.747125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae25440-ba5d-42f5-b31a-d3324a4e12aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:30.748726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dae25440-ba5d-42f5-b31a-d3324a4e12aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:30.858768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae25440-ba5d-42f5-b31a-d3324a4e12aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:30.860236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae25440-ba5d-42f5-b31a-d3324a4e12aa service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:30.878341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 719/1449] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:33.054990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43cc0f-4f4d-4f58-9740-c0d879d1be09 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:33.055523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43cc0f-4f4d-4f58-9740-c0d879d1be09 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:33.055523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43cc0f-4f4d-4f58-9740-c0d879d1be09 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:33.056901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 731/1450] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:33.064245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3742d7e-5ee2-4a31-8bc6-ada10b5f9ba6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:33.064666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3742d7e-5ee2-4a31-8bc6-ada10b5f9ba6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:33.064846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3742d7e-5ee2-4a31-8bc6-ada10b5f9ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:33.144367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a3742d7e-5ee2-4a31-8bc6-ada10b5f9ba6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:33.145588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a3742d7e-5ee2-4a31-8bc6-ada10b5f9ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:33.168150 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:33.249618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 720/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:33.292597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b43cc0f-4f4d-4f58-9740-c0d879d1be09 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:33.294276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5b43cc0f-4f4d-4f58-9740-c0d879d1be09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:33.405563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b43cc0f-4f4d-4f58-9740-c0d879d1be09 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:33.425651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:33.482897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543050a2-7cba-44d0-8874-6a05bbf513ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:33.483537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543050a2-7cba-44d0-8874-6a05bbf513ab service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:33.483813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543050a2-7cba-44d0-8874-6a05bbf513ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:33.484081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543050a2-7cba-44d0-8874-6a05bbf513ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:33.485694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-543050a2-7cba-44d0-8874-6a05bbf513ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:33.599645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543050a2-7cba-44d0-8874-6a05bbf513ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:33.600996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543050a2-7cba-44d0-8874-6a05bbf513ab service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:33.619142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 732/1452] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:35.723786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ad3ae-3fe9-4f5e-8aee-4bee7e012cef None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:35.724705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ad3ae-3fe9-4f5e-8aee-4bee7e012cef None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:35.724705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62ad3ae-3fe9-4f5e-8aee-4bee7e012cef None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:35.726927 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 721/1453] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:35.734092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02aa0b0-1d94-4536-b937-38e963700b36 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:35.734506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02aa0b0-1d94-4536-b937-38e963700b36 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:35.734648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02aa0b0-1d94-4536-b937-38e963700b36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:35.811156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:35.814262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b02aa0b0-1d94-4536-b937-38e963700b36 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:35.816310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b02aa0b0-1d94-4536-b937-38e963700b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:35.924380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 733/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:35.965819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62ad3ae-3fe9-4f5e-8aee-4bee7e012cef None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:35.967375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a62ad3ae-3fe9-4f5e-8aee-4bee7e012cef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:36.083193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62ad3ae-3fe9-4f5e-8aee-4bee7e012cef None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:36.135908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:36.162465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4856a1-332b-4941-971a-129b7d4ac1af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:36.163125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4856a1-332b-4941-971a-129b7d4ac1af service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:36.163402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4856a1-332b-4941-971a-129b7d4ac1af service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:36.163680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4856a1-332b-4941-971a-129b7d4ac1af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:36.165300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9e4856a1-332b-4941-971a-129b7d4ac1af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:36.287931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4856a1-332b-4941-971a-129b7d4ac1af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:36.289319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4856a1-332b-4941-971a-129b7d4ac1af service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:36.317460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 722/1455] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:38.333295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e863823f-c54f-4b7b-a7c7-23f9b60896ce None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:38.334115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e863823f-c54f-4b7b-a7c7-23f9b60896ce None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:38.334115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e863823f-c54f-4b7b-a7c7-23f9b60896ce None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:38.336760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 734/1456] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:38.344285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc0836-3f99-4f68-b782-a0290e368998 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:38.344766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc0836-3f99-4f68-b782-a0290e368998 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:38.344895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cc0836-3f99-4f68-b782-a0290e368998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:38.435167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-49cc0836-3f99-4f68-b782-a0290e368998 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:38.437016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-49cc0836-3f99-4f68-b782-a0290e368998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:38.470435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:38.540086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 723/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:04:38.577899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e863823f-c54f-4b7b-a7c7-23f9b60896ce None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:38.579470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e863823f-c54f-4b7b-a7c7-23f9b60896ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:38.687453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e863823f-c54f-4b7b-a7c7-23f9b60896ce None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:38.728584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:38.782467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f4fac1-226d-488b-9e81-192cbee56c25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:38.783132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f4fac1-226d-488b-9e81-192cbee56c25 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:38.783405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f4fac1-226d-488b-9e81-192cbee56c25 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:38.783674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f4fac1-226d-488b-9e81-192cbee56c25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:38.785285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-43f4fac1-226d-488b-9e81-192cbee56c25 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:38.912124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f4fac1-226d-488b-9e81-192cbee56c25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:38.913568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f4fac1-226d-488b-9e81-192cbee56c25 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:38.932325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 735/1458] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:41.012408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fc360c-94bf-457f-aa22-d24c589255a4 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:41.012966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fc360c-94bf-457f-aa22-d24c589255a4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:41.012966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fc360c-94bf-457f-aa22-d24c589255a4 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:41.014325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 724/1459] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:41.021820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842417bb-6104-46a4-83a7-9c12e3e3c80a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:41.022276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842417bb-6104-46a4-83a7-9c12e3e3c80a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:41.022371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842417bb-6104-46a4-83a7-9c12e3e3c80a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:41.091452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-842417bb-6104-46a4-83a7-9c12e3e3c80a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:41.092655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-842417bb-6104-46a4-83a7-9c12e3e3c80a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:41.101239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:41.193744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 736/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:41.239777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fc360c-94bf-457f-aa22-d24c589255a4 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:41.241299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-19fc360c-94bf-457f-aa22-d24c589255a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:41.355442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fc360c-94bf-457f-aa22-d24c589255a4 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:41.411466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:41.430643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76a4042-2455-4cd5-9d9f-741fcf456498 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:41.431295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a4042-2455-4cd5-9d9f-741fcf456498 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:41.431569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a4042-2455-4cd5-9d9f-741fcf456498 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:41.431838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76a4042-2455-4cd5-9d9f-741fcf456498 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:41.433482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c76a4042-2455-4cd5-9d9f-741fcf456498 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:41.545214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76a4042-2455-4cd5-9d9f-741fcf456498 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:41.546650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76a4042-2455-4cd5-9d9f-741fcf456498 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:41.564619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 725/1461] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:43.602577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eca52e-fbe5-4041-aa80-2668c096cdb0 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:43.603547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eca52e-fbe5-4041-aa80-2668c096cdb0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:43.603547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eca52e-fbe5-4041-aa80-2668c096cdb0 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:43.605678 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 737/1462] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:43.614742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba727e8e-4c18-4bba-82dd-cc1803e94968 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:43.615546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba727e8e-4c18-4bba-82dd-cc1803e94968 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:43.615657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba727e8e-4c18-4bba-82dd-cc1803e94968 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:43.689206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ba727e8e-4c18-4bba-82dd-cc1803e94968 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:43.690435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ba727e8e-4c18-4bba-82dd-cc1803e94968 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:43.740088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:43.800851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 726/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:43.838454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eca52e-fbe5-4041-aa80-2668c096cdb0 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:43.840132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-09eca52e-fbe5-4041-aa80-2668c096cdb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:43.949062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eca52e-fbe5-4041-aa80-2668c096cdb0 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:43.991596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:44.029421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d15de4f-0751-4ff2-8ae5-3d3adb91265e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:44.030044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15de4f-0751-4ff2-8ae5-3d3adb91265e service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:44.030328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15de4f-0751-4ff2-8ae5-3d3adb91265e service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:44.030597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15de4f-0751-4ff2-8ae5-3d3adb91265e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:44.032282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0d15de4f-0751-4ff2-8ae5-3d3adb91265e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:44.150022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d15de4f-0751-4ff2-8ae5-3d3adb91265e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:44.151497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d15de4f-0751-4ff2-8ae5-3d3adb91265e service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:44.171676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 738/1464] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:04:46.158974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67d1ae1-6daf-4ca1-af2c-ef6b2b4719cb None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:46.159567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67d1ae1-6daf-4ca1-af2c-ef6b2b4719cb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:46.159567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67d1ae1-6daf-4ca1-af2c-ef6b2b4719cb None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:46.160908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 727/1465] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:46.167569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6aef49-e229-435e-9940-3337b22f2127 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:46.168387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6aef49-e229-435e-9940-3337b22f2127 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:46.168493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6aef49-e229-435e-9940-3337b22f2127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:46.240878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4a6aef49-e229-435e-9940-3337b22f2127 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:46.242118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4a6aef49-e229-435e-9940-3337b22f2127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:46.260404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:46.350970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 739/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:46.387851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67d1ae1-6daf-4ca1-af2c-ef6b2b4719cb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:46.390090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d67d1ae1-6daf-4ca1-af2c-ef6b2b4719cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:46.503714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67d1ae1-6daf-4ca1-af2c-ef6b2b4719cb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:46.574706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:46.582489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f15870-143b-4f5b-830b-1237c46f00af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:46.583186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f15870-143b-4f5b-830b-1237c46f00af service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:46.583497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f15870-143b-4f5b-830b-1237c46f00af service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:46.583768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f15870-143b-4f5b-830b-1237c46f00af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:46.585394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-34f15870-143b-4f5b-830b-1237c46f00af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:46.696719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f15870-143b-4f5b-830b-1237c46f00af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:46.698159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f15870-143b-4f5b-830b-1237c46f00af service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:46.716767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 728/1467] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:48.706355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbb7cf-f953-49bb-b5b4-fcad83862977 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:48.706914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbb7cf-f953-49bb-b5b4-fcad83862977 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:48.706914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fbb7cf-f953-49bb-b5b4-fcad83862977 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:48.708182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 740/1468] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:04:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:04:48.714231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21198fb-0360-46d5-a067-118c4fe8843d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:48.714688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21198fb-0360-46d5-a067-118c4fe8843d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:48.714801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21198fb-0360-46d5-a067-118c4fe8843d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:48.794067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a21198fb-0360-46d5-a067-118c4fe8843d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:04:48.795339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a21198fb-0360-46d5-a067-118c4fe8843d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:48.848951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:48.902976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 729/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:04:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:04:48.941541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fbb7cf-f953-49bb-b5b4-fcad83862977 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:04:48.943147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-81fbb7cf-f953-49bb-b5b4-fcad83862977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:49.060904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fbb7cf-f953-49bb-b5b4-fcad83862977 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:04:49.113699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:49.143870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e57e92e-66e7-4c82-9735-526989713505 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:04:49.144492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e57e92e-66e7-4c82-9735-526989713505 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:04:49.144772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e57e92e-66e7-4c82-9735-526989713505 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:04:49.145050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e57e92e-66e7-4c82-9735-526989713505 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:04:49.146681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3e57e92e-66e7-4c82-9735-526989713505 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:04:49.269268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e57e92e-66e7-4c82-9735-526989713505 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:04:49.270638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e57e92e-66e7-4c82-9735-526989713505 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:04:49.290155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 741/1470] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:04:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:04:58.901064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=22 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:04:59.287794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=66 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:09.276333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ff9e0-6b25-492c-8424-4f490b92c66a None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:09.276892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ff9e0-6b25-492c-8424-4f490b92c66a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:09.276892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864ff9e0-6b25-492c-8424-4f490b92c66a None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:09.278197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 730/1471] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:05:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:09.284603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbebec75-f83a-4e19-94a6-cfe9f38c70b5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:09.285282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbebec75-f83a-4e19-94a6-cfe9f38c70b5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:09.285525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbebec75-f83a-4e19-94a6-cfe9f38c70b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:09.358568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-dbebec75-f83a-4e19-94a6-cfe9f38c70b5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:09.359814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dbebec75-f83a-4e19-94a6-cfe9f38c70b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:09.476595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 742/1472] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:05:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:05:09.566722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e4161-5cbc-46fa-a5e0-5277aec5df87 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:09.567112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e4161-5cbc-46fa-a5e0-5277aec5df87 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:09.567388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e4161-5cbc-46fa-a5e0-5277aec5df87 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:09.568783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 731/1473] 10.222.0.42 () {54 vars in 966 bytes} [Tue Jul 26 03:05:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:09.573732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badf7195-a728-45b3-9a7a-d1097348e9c9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:09.574014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badf7195-a728-45b3-9a7a-d1097348e9c9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:09.574288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badf7195-a728-45b3-9a7a-d1097348e9c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:09.649699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-badf7195-a728-45b3-9a7a-d1097348e9c9 None None] MFA Rules not processed for user `81ec7fc0b48643739402efba1285c713`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:09.650885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-badf7195-a728-45b3-9a7a-d1097348e9c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:09.673382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:09.732590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 743/1474] 10.222.0.42 () {58 vars in 1063 bytes} [Tue Jul 26 03:05:09 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:05:09.743975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888e4161-5cbc-46fa-a5e0-5277aec5df87 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:05:09.745532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-888e4161-5cbc-46fa-a5e0-5277aec5df87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:09.864092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888e4161-5cbc-46fa-a5e0-5277aec5df87 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:05:09.939904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c801d71f-101e-439d-9b57-bd709266eb13 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:05:09.940504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c801d71f-101e-439d-9b57-bd709266eb13 service glance] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:09.940786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c801d71f-101e-439d-9b57-bd709266eb13 service glance] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:09.941065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c801d71f-101e-439d-9b57-bd709266eb13 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:09.942700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c801d71f-101e-439d-9b57-bd709266eb13 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:10.000167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:10.061097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c801d71f-101e-439d-9b57-bd709266eb13 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:05:10.062549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c801d71f-101e-439d-9b57-bd709266eb13 service glance] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:05:10.082593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 732/1475] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:05:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:05:12.145941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1526785-7c88-4b00-8a1e-2ce3494d9bf4 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:12.146791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1526785-7c88-4b00-8a1e-2ce3494d9bf4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:12.146791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1526785-7c88-4b00-8a1e-2ce3494d9bf4 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:12.149035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 744/1476] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:05:12 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:12.154522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeedb952-ed2f-4df9-89cf-012364d88b49 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:12.154941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeedb952-ed2f-4df9-89cf-012364d88b49 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:12.155076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeedb952-ed2f-4df9-89cf-012364d88b49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:12.228230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-aeedb952-ed2f-4df9-89cf-012364d88b49 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:12.230054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-aeedb952-ed2f-4df9-89cf-012364d88b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:12.329112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 733/1477] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:05:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:05:12.352671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1526785-7c88-4b00-8a1e-2ce3494d9bf4 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:05:12.354182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c1526785-7c88-4b00-8a1e-2ce3494d9bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:12.490489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1526785-7c88-4b00-8a1e-2ce3494d9bf4 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:05:12.539685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:12.590166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7808b469-db60-4140-ab9a-bbd3ea3e0a5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:05:12.590717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808b469-db60-4140-ab9a-bbd3ea3e0a5a admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:12.590988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808b469-db60-4140-ab9a-bbd3ea3e0a5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:12.591298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808b469-db60-4140-ab9a-bbd3ea3e0a5a admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:12.592009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7808b469-db60-4140-ab9a-bbd3ea3e0a5a admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:05:12.593758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-7808b469-db60-4140-ab9a-bbd3ea3e0a5a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:05:12.593758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:05:12.593758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:05:12.594913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7808b469-db60-4140-ab9a-bbd3ea3e0a5a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:05:12.614437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 745/1478] 10.222.0.42 () {56 vars in 1108 bytes} [Tue Jul 26 03:05:12 2022] GET /identity/v3/projects => generated 2553 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:05:14.606980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4b94e2-e2a2-4df8-a577-0e04f2026871 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:14.607860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4b94e2-e2a2-4df8-a577-0e04f2026871 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:14.607860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4b94e2-e2a2-4df8-a577-0e04f2026871 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:14.610101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 734/1479] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:05:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:14.615567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160358e2-0a4d-49d1-8735-527cadf05f3c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:14.616028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160358e2-0a4d-49d1-8735-527cadf05f3c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:14.616135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160358e2-0a4d-49d1-8735-527cadf05f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:14.683945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-160358e2-0a4d-49d1-8735-527cadf05f3c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:14.685151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-160358e2-0a4d-49d1-8735-527cadf05f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:14.784617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 746/1480] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:05:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:05:14.835286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4b94e2-e2a2-4df8-a577-0e04f2026871 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:05:14.837204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ba4b94e2-e2a2-4df8-a577-0e04f2026871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:14.847527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:14.989609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4b94e2-e2a2-4df8-a577-0e04f2026871 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:05:15.075595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:15.108935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2840a3b7-f5b4-4469-ac07-7b81dcbabca2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:05:15.109563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2840a3b7-f5b4-4469-ac07-7b81dcbabca2 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:15.109843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2840a3b7-f5b4-4469-ac07-7b81dcbabca2 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:15.110120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2840a3b7-f5b4-4469-ac07-7b81dcbabca2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:15.111796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2840a3b7-f5b4-4469-ac07-7b81dcbabca2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:15.212981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2840a3b7-f5b4-4469-ac07-7b81dcbabca2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:05:15.214593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2840a3b7-f5b4-4469-ac07-7b81dcbabca2 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:05:15.233087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 735/1481] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:05:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:05:17.412006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67debcb-c1ba-4b7e-a612-ba19098e8743 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:17.412584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67debcb-c1ba-4b7e-a612-ba19098e8743 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:17.412584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67debcb-c1ba-4b7e-a612-ba19098e8743 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:17.414714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 747/1482] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:05:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:17.421461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd64d48-f7db-442f-a215-f60d62ae12ff None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:17.421932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd64d48-f7db-442f-a215-f60d62ae12ff None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:17.422027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd64d48-f7db-442f-a215-f60d62ae12ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:17.495763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-bfd64d48-f7db-442f-a215-f60d62ae12ff None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:17.497664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bfd64d48-f7db-442f-a215-f60d62ae12ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:17.551346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:17.606526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 736/1483] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:05:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:05:17.645875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67debcb-c1ba-4b7e-a612-ba19098e8743 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:05:17.647469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b67debcb-c1ba-4b7e-a612-ba19098e8743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:17.672758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:17.758324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67debcb-c1ba-4b7e-a612-ba19098e8743 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:05:17.841518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b3e85b-7499-478a-b137-88844a83f0fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:05:17.842142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b3e85b-7499-478a-b137-88844a83f0fa service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:17.842417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b3e85b-7499-478a-b137-88844a83f0fa service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:17.842681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b3e85b-7499-478a-b137-88844a83f0fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:17.844352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-78b3e85b-7499-478a-b137-88844a83f0fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:17.920480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:17.956645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b3e85b-7499-478a-b137-88844a83f0fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:05:17.958030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b3e85b-7499-478a-b137-88844a83f0fa service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:05:17.978977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 748/1484] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:05:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:05:20.005450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6d6c3-8735-4d40-95eb-b45b3de773f7 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:20.006371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6d6c3-8735-4d40-95eb-b45b3de773f7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:20.006371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6d6c3-8735-4d40-95eb-b45b3de773f7 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:20.008519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 737/1485] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:05:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:20.014230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391a16a9-ddc8-4b38-92a3-77a91a26057a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:20.015270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391a16a9-ddc8-4b38-92a3-77a91a26057a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:20.015270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391a16a9-ddc8-4b38-92a3-77a91a26057a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:20.082975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-391a16a9-ddc8-4b38-92a3-77a91a26057a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:20.084196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-391a16a9-ddc8-4b38-92a3-77a91a26057a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:20.185388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 749/1486] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:05:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:05:20.228135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6d6c3-8735-4d40-95eb-b45b3de773f7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:05:20.230594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-75b6d6c3-8735-4d40-95eb-b45b3de773f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:20.347181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6d6c3-8735-4d40-95eb-b45b3de773f7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:05:20.432089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:20.434409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560588ae-812c-4000-b0f2-c478b96506a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:05:20.435114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560588ae-812c-4000-b0f2-c478b96506a5 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:20.435373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560588ae-812c-4000-b0f2-c478b96506a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:20.435652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560588ae-812c-4000-b0f2-c478b96506a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:20.437956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-560588ae-812c-4000-b0f2-c478b96506a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:20.560800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560588ae-812c-4000-b0f2-c478b96506a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:05:20.562928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560588ae-812c-4000-b0f2-c478b96506a5 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:05:20.586327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 738/1487] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:05:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:05:22.675809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840a9fda-9355-441f-9426-20062874c797 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:22.676404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840a9fda-9355-441f-9426-20062874c797 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:22.676404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840a9fda-9355-441f-9426-20062874c797 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:22.677765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 750/1488] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:05:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:22.686639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6393667-6df0-4934-ac79-8cd53022afec None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:22.687412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6393667-6df0-4934-ac79-8cd53022afec None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:22.687692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6393667-6df0-4934-ac79-8cd53022afec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:22.763211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d6393667-6df0-4934-ac79-8cd53022afec None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:22.764369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d6393667-6df0-4934-ac79-8cd53022afec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:22.826476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:22.873031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 739/1489] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:05:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:05:22.913784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840a9fda-9355-441f-9426-20062874c797 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:05:22.916092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-840a9fda-9355-441f-9426-20062874c797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:22.956827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:23.034233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840a9fda-9355-441f-9426-20062874c797 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:05:23.112970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008299b3-a711-40ad-bf10-27a59fcd1193 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:05:23.113600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008299b3-a711-40ad-bf10-27a59fcd1193 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:23.113879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008299b3-a711-40ad-bf10-27a59fcd1193 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:23.114151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008299b3-a711-40ad-bf10-27a59fcd1193 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:23.115839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-008299b3-a711-40ad-bf10-27a59fcd1193 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:23.198612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:23.228776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008299b3-a711-40ad-bf10-27a59fcd1193 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:05:23.230235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008299b3-a711-40ad-bf10-27a59fcd1193 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:05:23.248344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 751/1490] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:05:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:05:25.538993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074fadc-0d33-4fe8-82c9-aa841c7a49f4 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:25.539882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074fadc-0d33-4fe8-82c9-aa841c7a49f4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:25.540055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b074fadc-0d33-4fe8-82c9-aa841c7a49f4 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:25.542270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 740/1491] 10.222.0.42 () {54 vars in 926 bytes} [Tue Jul 26 03:05:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:05:25.548751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a88875-5335-4785-b16c-982de08315cc None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:25.549207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a88875-5335-4785-b16c-982de08315cc None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:25.549317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a88875-5335-4785-b16c-982de08315cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:25.623787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-31a88875-5335-4785-b16c-982de08315cc None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:05:25.625000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-31a88875-5335-4785-b16c-982de08315cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:25.729059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 752/1492] 10.222.0.42 () {58 vars in 1023 bytes} [Tue Jul 26 03:05:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:05:25.768854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074fadc-0d33-4fe8-82c9-aa841c7a49f4 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:05:25.770429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b074fadc-0d33-4fe8-82c9-aa841c7a49f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:25.881296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b074fadc-0d33-4fe8-82c9-aa841c7a49f4 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:05:25.957305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab5e97-1db5-44b2-9213-123149eef6f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:05:25.957921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab5e97-1db5-44b2-9213-123149eef6f4 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:05:25.958200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab5e97-1db5-44b2-9213-123149eef6f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:05:25.958475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab5e97-1db5-44b2-9213-123149eef6f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:05:25.960122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9aab5e97-1db5-44b2-9213-123149eef6f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:05:25.969776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:26.092349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aab5e97-1db5-44b2-9213-123149eef6f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:05:26.093808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aab5e97-1db5-44b2-9213-123149eef6f4 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:05:26.114086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 741/1493] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:05:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:05:35.725505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=82 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:05:36.111206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=50 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:54.426992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:54.427925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:54.427925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:54.511142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:06:54.512478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:54.616252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 753/1494] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:06:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:06:54.625514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c555ed-9e6c-4108-a505-cc33a3d0c3de None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:54.626023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c555ed-9e6c-4108-a505-cc33a3d0c3de None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:54.626127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c555ed-9e6c-4108-a505-cc33a3d0c3de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:54.717439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a7c555ed-9e6c-4108-a505-cc33a3d0c3de None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:06:54.718572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a7c555ed-9e6c-4108-a505-cc33a3d0c3de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:54.817310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 742/1495] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:06:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:06:54.822203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:54.823492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:54.835920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8d1060-4f0e-4743-b71c-640cbd52fb25 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:54.901534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:54.938750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:54.939414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:54.939700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:54.939977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:54.940719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:06:54.941021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:54.942082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:54.942082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:06:54.942082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:06:54.942995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:54.955036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 754/1496] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:06:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:06:54.964070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c555ed-9e6c-4108-a505-cc33a3d0c3de None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:54.973895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c555ed-9e6c-4108-a505-cc33a3d0c3de None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:55.038154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:55.073148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:55.073788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:55.074066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:55.074340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:55.075056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:55.076811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING py.warnings [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:55.076811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: warnings.warn(msg) Jul 26 03:06:55.076811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: Jul 26 03:06:55.078052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:55.136957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 743/1497] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:06:54 2022] POST /identity/v3/projects => generated 379 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:06:55.141629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:55.143237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:55.179606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3d2251-1448-4e5e-93ab-94032ba63ee5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:55.280620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:55.281162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:55.281444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:55.281712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:55.282380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:55.283436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:55.283436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:06:55.283436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:06:55.284655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:55.330442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 755/1498] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:06:55 2022] POST /identity/v3/users => generated 290 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:06:55.335137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:55.336717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:55.402258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1822b079-a428-45b9-af66-04f2c807dac7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:55.491824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:55.504596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:55.505230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:55.505505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:55.505775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:55.506482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:55.507578 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING py.warnings [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:55.507578 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: warnings.warn(msg) Jul 26 03:06:55.507578 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: Jul 26 03:06:55.508476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:55.521641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 744/1499] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:06:55.525902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:55.527517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:55.539363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ff62d2-f27f-4fb3-bd9e-0533bfd22882 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:55.566032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:55.644823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deb192e-064f-4adb-9ef7-25be2db9637c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:55.645445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb192e-064f-4adb-9ef7-25be2db9637c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:55.645724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb192e-064f-4adb-9ef7-25be2db9637c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:55.645993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb192e-064f-4adb-9ef7-25be2db9637c admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:55.646699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deb192e-064f-4adb-9ef7-25be2db9637c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:55.647825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-7deb192e-064f-4adb-9ef7-25be2db9637c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:55.647825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:06:55.647825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:06:55.648747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deb192e-064f-4adb-9ef7-25be2db9637c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:55.661564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 756/1500] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:06:55.666373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:55.667959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:55.702082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40b0403-0b95-471a-90f3-0faf6cbb6ed6 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:55.806045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e9cb03-6ffb-4c94-b131-f03bf45e2f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:55.806643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e9cb03-6ffb-4c94-b131-f03bf45e2f47 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:55.806919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e9cb03-6ffb-4c94-b131-f03bf45e2f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:55.807309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e9cb03-6ffb-4c94-b131-f03bf45e2f47 admin admin] PATH_INFO: `/v3/projects/556a66d7000343778feceab8b23229fc/users/5f12ff6c2bd6404f9bf69e035c3d7702/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:55.830575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e9cb03-6ffb-4c94-b131-f03bf45e2f47 admin admin] RBAC: Authorizing `identity:create_grant(project_id=556a66d7000343778feceab8b23229fc, user_id=5f12ff6c2bd6404f9bf69e035c3d7702, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:55.831661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING py.warnings [None req-79e9cb03-6ffb-4c94-b131-f03bf45e2f47 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:55.831661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: warnings.warn(msg) Jul 26 03:06:55.831661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: Jul 26 03:06:55.833952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e9cb03-6ffb-4c94-b131-f03bf45e2f47 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:55.860561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 745/1501] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:06:55 2022] PUT /identity/v3/projects/556a66d7000343778feceab8b23229fc/users/5f12ff6c2bd6404f9bf69e035c3d7702/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:06:55.874134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9cc09e-49f6-4ba2-acb9-a41261229e4e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:55.874472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9cc09e-49f6-4ba2-acb9-a41261229e4e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:55.875191 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9cc09e-49f6-4ba2-acb9-a41261229e4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:55.942772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ab9cc09e-49f6-4ba2-acb9-a41261229e4e None None] MFA Rules not processed for user `5f12ff6c2bd6404f9bf69e035c3d7702`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:06:55.944041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ab9cc09e-49f6-4ba2-acb9-a41261229e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:56.026542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 757/1502] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:06:55 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:06:56.040122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfb96d3-6d4d-41eb-8d4d-ba51ef77e13c None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:56.040410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfb96d3-6d4d-41eb-8d4d-ba51ef77e13c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:56.040678 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfb96d3-6d4d-41eb-8d4d-ba51ef77e13c None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:56.042055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 746/1503] 10.222.0.42 () {54 vars in 964 bytes} [Tue Jul 26 03:06:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:06:56.047245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd26b40-6954-45c3-992a-6c1e4bac233b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:56.047531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd26b40-6954-45c3-992a-6c1e4bac233b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:56.047806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd26b40-6954-45c3-992a-6c1e4bac233b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:56.080307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:56.125573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-dfd26b40-6954-45c3-992a-6c1e4bac233b None None] MFA Rules not processed for user `48f06347846c48c2933cee7ebd32b947`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:06:56.126791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dfd26b40-6954-45c3-992a-6c1e4bac233b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:56.222736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 758/1504] 10.222.0.42 () {58 vars in 1061 bytes} [Tue Jul 26 03:06:56 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:06:56.231133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfb96d3-6d4d-41eb-8d4d-ba51ef77e13c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:56.232665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2cfb96d3-6d4d-41eb-8d4d-ba51ef77e13c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:56.285811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:56.344375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfb96d3-6d4d-41eb-8d4d-ba51ef77e13c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:56.425824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:56.426451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:56.426729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:56.426997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:56.428656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:56.539163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:56.540620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:56.544289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:56.559573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 747/1505] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:06:56 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:06:56.608298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c10f31-e916-458b-9d1b-4efe619aede7 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:56.608756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c10f31-e916-458b-9d1b-4efe619aede7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:56.608853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c10f31-e916-458b-9d1b-4efe619aede7 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:56.610128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 759/1506] 10.222.0.42 () {52 vars in 830 bytes} [Tue Jul 26 03:06:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:06:56.614103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:56.615364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:56.729984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1cd6aa-f3c5-4fc1-8d96-75a857112a5e service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:56.829640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:56.830292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:56.830569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:56.830842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] PATH_INFO: `/v3/users/5f12ff6c2bd6404f9bf69e035c3d7702` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:56.845565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f12ff6c2bd6404f9bf69e035c3d7702)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:56.846542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING py.warnings [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:56.846542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: warnings.warn(msg) Jul 26 03:06:56.846542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: Jul 26 03:06:56.847803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:56.859277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:56.923045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f12ff6c2bd6404f9bf69e035c3d7702'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:56.935846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f12ff6c2bd6404f9bf69e035c3d7702'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:56.946783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f12ff6c2bd6404f9bf69e035c3d7702'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:56.966294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f12ff6c2bd6404f9bf69e035c3d7702'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:56.966722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f12ff6c2bd6404f9bf69e035c3d7702'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:56.973920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 748/1507] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:06:56 2022] DELETE /identity/v3/users/5f12ff6c2bd6404f9bf69e035c3d7702 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:06:56.978676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:56.978983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:56.979275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:57.051137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:06:57.052400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:57.137114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:57.150219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 760/1508] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:06:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:06:57.160893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:57.162131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:57.262277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca599fb-a63d-4949-8b54-1c22845c7bf9 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:57.317805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:06:57.339790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dabb03e-754c-4f9e-872c-814de734c02d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:57.340427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dabb03e-754c-4f9e-872c-814de734c02d service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:57.340708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dabb03e-754c-4f9e-872c-814de734c02d service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:57.340982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dabb03e-754c-4f9e-872c-814de734c02d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:57.342609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6dabb03e-754c-4f9e-872c-814de734c02d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:57.453215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dabb03e-754c-4f9e-872c-814de734c02d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:57.454582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dabb03e-754c-4f9e-872c-814de734c02d service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:57.473592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 749/1509] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:06:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:06:57.834851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:06:57.836158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:06:57.848505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a913c1b6-7ef9-48e1-bf6b-c27236003443 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:06:57.953219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:06:57.953905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:06:57.954184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:06:57.954461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] PATH_INFO: `/v3/projects/556a66d7000343778feceab8b23229fc` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:06:57.965933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=556a66d7000343778feceab8b23229fc)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:06:57.966903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:06:57.966903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:06:57.966903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:06:57.968147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:06:58.032792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '556a66d7000343778feceab8b23229fc', 'request_id': 'req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:58.041958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '556a66d7000343778feceab8b23229fc', 'request_id': 'req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:58.048067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '556a66d7000343778feceab8b23229fc', 'request_id': 'req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:58.077759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] The token cache is being invalidate because project 556a66d7000343778feceab8b23229fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:06:58.078046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:58.078473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ce7cf0b0-6474-4d87-ac4e-8ef0dffe10e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:06:58.079985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 761/1510] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:06:57 2022] DELETE /identity/v3/projects/556a66d7000343778feceab8b23229fc => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:07:07.471204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=62 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:07:08.075702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:08:34.295585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9520bf95-32bd-4a3d-aa50-300794bc0505 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:34.296182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9520bf95-32bd-4a3d-aa50-300794bc0505 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:34.296182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9520bf95-32bd-4a3d-aa50-300794bc0505 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:34.298047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 750/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:08:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:34.303212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be9a9a-b5b0-487b-bc85-d95b848997c2 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:34.303673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be9a9a-b5b0-487b-bc85-d95b848997c2 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:34.303779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70be9a9a-b5b0-487b-bc85-d95b848997c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:34.393611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-70be9a9a-b5b0-487b-bc85-d95b848997c2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:34.394812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-70be9a9a-b5b0-487b-bc85-d95b848997c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:34.508378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 762/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:08:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:34.662727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9520bf95-32bd-4a3d-aa50-300794bc0505 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:34.666244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9520bf95-32bd-4a3d-aa50-300794bc0505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:34.803821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9520bf95-32bd-4a3d-aa50-300794bc0505 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:34.810227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad510ca-0022-4e24-ba1a-c1fbc15359ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:34.810866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad510ca-0022-4e24-ba1a-c1fbc15359ec service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:34.811184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad510ca-0022-4e24-ba1a-c1fbc15359ec service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:34.811462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad510ca-0022-4e24-ba1a-c1fbc15359ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:34.820222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad510ca-0022-4e24-ba1a-c1fbc15359ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:34.821804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad510ca-0022-4e24-ba1a-c1fbc15359ec service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:34.831410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 751/1513] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:08:35.661750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d66a3-6db6-433e-a8d2-8b04bc6646d8 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:35.662588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d66a3-6db6-433e-a8d2-8b04bc6646d8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:35.662588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5d66a3-6db6-433e-a8d2-8b04bc6646d8 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:35.664917 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 763/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:08:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:35.669865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ac095-f890-49b8-8ab3-60264dbf6245 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:35.670311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ac095-f890-49b8-8ab3-60264dbf6245 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:35.670424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ac095-f890-49b8-8ab3-60264dbf6245 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:35.692041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-031ac095-f890-49b8-8ab3-60264dbf6245 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:35.693275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-031ac095-f890-49b8-8ab3-60264dbf6245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:35.702117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 752/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:08:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:35.852241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d66a3-6db6-433e-a8d2-8b04bc6646d8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:35.869762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5d66a3-6db6-433e-a8d2-8b04bc6646d8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:35.876775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4303b5fd-e1f4-4b6b-a44d-1859000a896b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:35.877421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303b5fd-e1f4-4b6b-a44d-1859000a896b service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:35.877706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303b5fd-e1f4-4b6b-a44d-1859000a896b service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:35.878078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4303b5fd-e1f4-4b6b-a44d-1859000a896b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:35.887197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4303b5fd-e1f4-4b6b-a44d-1859000a896b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:35.889683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4303b5fd-e1f4-4b6b-a44d-1859000a896b service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:35.902486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 764/1516] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:08:36.506150 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5d942-a506-4bd5-b656-72c6625056ab None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:36.506392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5d942-a506-4bd5-b656-72c6625056ab None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:36.506726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c5d942-a506-4bd5-b656-72c6625056ab None None] PATH_INFO: `/v3` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:36.508101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 753/1517] 10.222.0.36 () {54 vars in 925 bytes} [Tue Jul 26 03:08:36 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:08:36.515613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69efead6-c83e-4cc0-9c89-b53b5464213b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:36.515898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69efead6-c83e-4cc0-9c89-b53b5464213b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:36.516178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69efead6-c83e-4cc0-9c89-b53b5464213b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:36.584942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-69efead6-c83e-4cc0-9c89-b53b5464213b None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:36.586155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-69efead6-c83e-4cc0-9c89-b53b5464213b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:36.603519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:08:36.659942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 765/1518] 10.222.0.36 () {58 vars in 1015 bytes} [Tue Jul 26 03:08:36 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:40.507683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6fb1eb-34a1-458e-85e2-535942becf7c None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:40.508253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6fb1eb-34a1-458e-85e2-535942becf7c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:40.508253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6fb1eb-34a1-458e-85e2-535942becf7c None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:40.509625 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 754/1519] 10.222.0.42 () {54 vars in 961 bytes} [Tue Jul 26 03:08:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:40.514585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e15d58-bfd4-4fa1-9e43-680a5d4f74a5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:40.515146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e15d58-bfd4-4fa1-9e43-680a5d4f74a5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:40.515267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e15d58-bfd4-4fa1-9e43-680a5d4f74a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:40.535531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-57e15d58-bfd4-4fa1-9e43-680a5d4f74a5 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:40.536857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-57e15d58-bfd4-4fa1-9e43-680a5d4f74a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:40.549651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 766/1520] 10.222.0.42 () {58 vars in 1058 bytes} [Tue Jul 26 03:08:40 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:40.553226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6fb1eb-34a1-458e-85e2-535942becf7c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:40.566870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6fb1eb-34a1-458e-85e2-535942becf7c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:40.571873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab3d8c8-aae2-4cb0-8099-5ad004cf44a0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:40.572499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab3d8c8-aae2-4cb0-8099-5ad004cf44a0 service placement] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:40.572778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab3d8c8-aae2-4cb0-8099-5ad004cf44a0 service placement] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:40.573082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab3d8c8-aae2-4cb0-8099-5ad004cf44a0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:40.582562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab3d8c8-aae2-4cb0-8099-5ad004cf44a0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:40.584193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab3d8c8-aae2-4cb0-8099-5ad004cf44a0 service placement] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:40.593924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 755/1521] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:40 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:08:47.145309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61934e52-da92-420a-a3c1-59f5fe79da1c None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:47.146244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61934e52-da92-420a-a3c1-59f5fe79da1c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:47.146244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61934e52-da92-420a-a3c1-59f5fe79da1c None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:47.148166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 767/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:08:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:47.156133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b931c90-929b-47f0-ac74-b79fafaffaae None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:47.156614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b931c90-929b-47f0-ac74-b79fafaffaae None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:47.156614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b931c90-929b-47f0-ac74-b79fafaffaae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:47.178635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3b931c90-929b-47f0-ac74-b79fafaffaae None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:47.179889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3b931c90-929b-47f0-ac74-b79fafaffaae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:47.188970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 756/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:08:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:47.345187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61934e52-da92-420a-a3c1-59f5fe79da1c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:47.361808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61934e52-da92-420a-a3c1-59f5fe79da1c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:47.372274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7fd058-8f7e-4066-a1ff-1bccff241240 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:47.373074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7fd058-8f7e-4066-a1ff-1bccff241240 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:47.373438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7fd058-8f7e-4066-a1ff-1bccff241240 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:47.373784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7fd058-8f7e-4066-a1ff-1bccff241240 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:47.394872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7fd058-8f7e-4066-a1ff-1bccff241240 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:47.397324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7fd058-8f7e-4066-a1ff-1bccff241240 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:47.411804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 768/1524] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:08:48.218048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a436b2-64ad-4815-bc4a-1c1efe18f598 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:48.219037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a436b2-64ad-4815-bc4a-1c1efe18f598 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:48.219037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a436b2-64ad-4815-bc4a-1c1efe18f598 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:48.221236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 757/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 26 03:08:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:48.226528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6407935e-b51f-487d-bfb0-81cd31dcaa05 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:48.227018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6407935e-b51f-487d-bfb0-81cd31dcaa05 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:48.227125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6407935e-b51f-487d-bfb0-81cd31dcaa05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:48.247548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-6407935e-b51f-487d-bfb0-81cd31dcaa05 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:48.248783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6407935e-b51f-487d-bfb0-81cd31dcaa05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:48.257828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 769/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 26 03:08:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:48.411010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a436b2-64ad-4815-bc4a-1c1efe18f598 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:48.428129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a436b2-64ad-4815-bc4a-1c1efe18f598 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:48.434492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61c4154-f174-48a2-8358-46c94b281ed2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:48.435140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61c4154-f174-48a2-8358-46c94b281ed2 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:48.435421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61c4154-f174-48a2-8358-46c94b281ed2 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:48.435698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61c4154-f174-48a2-8358-46c94b281ed2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:48.444593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61c4154-f174-48a2-8358-46c94b281ed2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:48.446191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61c4154-f174-48a2-8358-46c94b281ed2 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:48.457328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 758/1527] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:08:52.158356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c227b32-7b24-4128-9362-4ea7f4993053 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:52.159008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c227b32-7b24-4128-9362-4ea7f4993053 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:52.159008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c227b32-7b24-4128-9362-4ea7f4993053 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:52.160304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 770/1528] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:08:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:52.166870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d893b514-5dd8-47a8-ab1c-ae70460b6281 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:52.167639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d893b514-5dd8-47a8-ab1c-ae70460b6281 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:52.167829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d893b514-5dd8-47a8-ab1c-ae70460b6281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:52.192262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d893b514-5dd8-47a8-ab1c-ae70460b6281 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:52.193502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d893b514-5dd8-47a8-ab1c-ae70460b6281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:52.201968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 759/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:08:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:52.276353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c227b32-7b24-4128-9362-4ea7f4993053 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:52.278499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4c227b32-7b24-4128-9362-4ea7f4993053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:52.365827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c227b32-7b24-4128-9362-4ea7f4993053 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:52.370078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97d1024-3dfd-41fb-bc3b-28a9297ab72d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:52.370659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d1024-3dfd-41fb-bc3b-28a9297ab72d service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:52.370983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d1024-3dfd-41fb-bc3b-28a9297ab72d service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:52.371277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d1024-3dfd-41fb-bc3b-28a9297ab72d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:52.380979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97d1024-3dfd-41fb-bc3b-28a9297ab72d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:52.382409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97d1024-3dfd-41fb-bc3b-28a9297ab72d service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:52.391976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 771/1530] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:08:54.354435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dde5503-2413-454e-8fa1-7bf608321419 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:54.355349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dde5503-2413-454e-8fa1-7bf608321419 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:54.355349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dde5503-2413-454e-8fa1-7bf608321419 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:54.357441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 760/1531] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:08:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:54.363034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899b063c-70ba-4d97-8bf8-b3b7ff90403a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:54.363484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899b063c-70ba-4d97-8bf8-b3b7ff90403a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:54.363579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899b063c-70ba-4d97-8bf8-b3b7ff90403a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:54.383656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-899b063c-70ba-4d97-8bf8-b3b7ff90403a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:54.384871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-899b063c-70ba-4d97-8bf8-b3b7ff90403a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:54.393455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 772/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:08:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:54.430160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dde5503-2413-454e-8fa1-7bf608321419 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:54.442475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dde5503-2413-454e-8fa1-7bf608321419 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:54.447057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367939a-3bce-4e76-9d9f-41c97fdf9c76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:54.447739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367939a-3bce-4e76-9d9f-41c97fdf9c76 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:54.448022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367939a-3bce-4e76-9d9f-41c97fdf9c76 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:54.448315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367939a-3bce-4e76-9d9f-41c97fdf9c76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:54.459258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367939a-3bce-4e76-9d9f-41c97fdf9c76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:54.460694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367939a-3bce-4e76-9d9f-41c97fdf9c76 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:54.470302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 761/1533] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:08:57.979511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c48632f-9fb2-4cd8-ae51-c8ea7477619c None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:57.980073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c48632f-9fb2-4cd8-ae51-c8ea7477619c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:57.980073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c48632f-9fb2-4cd8-ae51-c8ea7477619c None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:57.981385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 773/1534] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:08:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:08:57.987537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476b0c2c-553f-45bd-95ea-40298f3a5151 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:57.987966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476b0c2c-553f-45bd-95ea-40298f3a5151 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:57.988059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476b0c2c-553f-45bd-95ea-40298f3a5151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:57.996613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=81 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:08:58.010715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-476b0c2c-553f-45bd-95ea-40298f3a5151 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:08:58.011972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-476b0c2c-553f-45bd-95ea-40298f3a5151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:58.020945 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 762/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:08:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:08:58.326320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c48632f-9fb2-4cd8-ae51-c8ea7477619c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:08:58.330313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7c48632f-9fb2-4cd8-ae51-c8ea7477619c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:08:58.407849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:08:58.427783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c48632f-9fb2-4cd8-ae51-c8ea7477619c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:08:58.433095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eab3083-6bd9-4770-a9db-6f5da6a7e5bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:08:58.433947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eab3083-6bd9-4770-a9db-6f5da6a7e5bc service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:08:58.434252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eab3083-6bd9-4770-a9db-6f5da6a7e5bc service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:08:58.434568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eab3083-6bd9-4770-a9db-6f5da6a7e5bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:08:58.447216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eab3083-6bd9-4770-a9db-6f5da6a7e5bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:08:58.448656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eab3083-6bd9-4770-a9db-6f5da6a7e5bc service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:08:58.458562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 774/1536] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:08:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:09:00.462903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e0f60-bc13-4963-ba41-5fc9acd6f086 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:09:00.463528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e0f60-bc13-4963-ba41-5fc9acd6f086 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:09:00.463528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e0f60-bc13-4963-ba41-5fc9acd6f086 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:09:00.464862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 763/1537] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 26 03:09:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:09:00.470942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da17a5e6-57b1-41a3-8c40-13e259aacb8c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:09:00.471437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da17a5e6-57b1-41a3-8c40-13e259aacb8c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:09:00.471532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da17a5e6-57b1-41a3-8c40-13e259aacb8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:09:00.499839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-da17a5e6-57b1-41a3-8c40-13e259aacb8c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:09:00.501709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-da17a5e6-57b1-41a3-8c40-13e259aacb8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:09:00.513667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 775/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 26 03:09:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:09:00.815619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4e0f60-bc13-4963-ba41-5fc9acd6f086 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:09:00.818387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8f4e0f60-bc13-4963-ba41-5fc9acd6f086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:09:00.833721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4e0f60-bc13-4963-ba41-5fc9acd6f086 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:09:00.837792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19148680-b772-4cb9-9178-25942fa59b80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:09:00.838418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19148680-b772-4cb9-9178-25942fa59b80 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:09:00.838705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19148680-b772-4cb9-9178-25942fa59b80 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:09:00.838979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19148680-b772-4cb9-9178-25942fa59b80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:09:00.849620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19148680-b772-4cb9-9178-25942fa59b80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:09:00.851127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19148680-b772-4cb9-9178-25942fa59b80 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:09:00.862072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 764/1539] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:09:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:09:10.494452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=11 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:09:10.857336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=8 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:30.464500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b572e635-1c80-4626-8203-bc07e6fbaae9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.465623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b572e635-1c80-4626-8203-bc07e6fbaae9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.466137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b572e635-1c80-4626-8203-bc07e6fbaae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.493496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b572e635-1c80-4626-8203-bc07e6fbaae9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:30.494741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b572e635-1c80-4626-8203-bc07e6fbaae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:30.509311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 776/1540] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:30.518800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2288c092-4486-4e38-8eaa-e7004b7250d0 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.519460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2288c092-4486-4e38-8eaa-e7004b7250d0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.519460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2288c092-4486-4e38-8eaa-e7004b7250d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.549414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2288c092-4486-4e38-8eaa-e7004b7250d0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:30.550678 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2288c092-4486-4e38-8eaa-e7004b7250d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:30.569236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 765/1541] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:30.576973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b572e635-1c80-4626-8203-bc07e6fbaae9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:30.587867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b572e635-1c80-4626-8203-bc07e6fbaae9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:30.599409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:30.600061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.600339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.600616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.601341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:30.601653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:30.603301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:30.621219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 777/1542] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:30.629537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2288c092-4486-4e38-8eaa-e7004b7250d0 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:30.645517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2288c092-4486-4e38-8eaa-e7004b7250d0 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:30.653172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bdc686-629c-4bcc-9e97-bcd16927414d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:30.653811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdc686-629c-4bcc-9e97-bcd16927414d admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.654088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdc686-629c-4bcc-9e97-bcd16927414d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.654361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdc686-629c-4bcc-9e97-bcd16927414d admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.655121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bdc686-629c-4bcc-9e97-bcd16927414d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:30.657033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bdc686-629c-4bcc-9e97-bcd16927414d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:30.726034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 766/1543] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/projects => generated 391 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:30.729934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:30.735871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee1f8f-f28c-4bd7-9eb4-861ceba3e76c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.736166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee1f8f-f28c-4bd7-9eb4-861ceba3e76c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.736442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee1f8f-f28c-4bd7-9eb4-861ceba3e76c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.746739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eafa128-3bea-4e85-bfc8-88e0fd4b3740 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:30.767266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f8ee1f8f-f28c-4bd7-9eb4-861ceba3e76c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:30.768494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f8ee1f8f-f28c-4bd7-9eb4-861ceba3e76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:30.825840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8389bf6e-bc9a-4de8-8884-8eb083abd0c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:30.826523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8389bf6e-bc9a-4de8-8884-8eb083abd0c0 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.826805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8389bf6e-bc9a-4de8-8884-8eb083abd0c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.827106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8389bf6e-bc9a-4de8-8884-8eb083abd0c0 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.827801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8389bf6e-bc9a-4de8-8884-8eb083abd0c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:30.830094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8389bf6e-bc9a-4de8-8884-8eb083abd0c0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:30.878790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 767/1544] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Jul 26 03:10:30.888991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e1edf-69d6-41cf-a908-3558ec20611f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.889330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e1edf-69d6-41cf-a908-3558ec20611f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.889637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e1edf-69d6-41cf-a908-3558ec20611f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.892135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 778/1545] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/users => generated 296 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:30.893489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9c47c8-5552-47d5-aa57-f3b7edb26806 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.893785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9c47c8-5552-47d5-aa57-f3b7edb26806 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.894064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9c47c8-5552-47d5-aa57-f3b7edb26806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.926458 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8e9e1edf-69d6-41cf-a908-3558ec20611f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:30.927725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8e9e1edf-69d6-41cf-a908-3558ec20611f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:30.933497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-7a9c47c8-5552-47d5-aa57-f3b7edb26806 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:30.934672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7a9c47c8-5552-47d5-aa57-f3b7edb26806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:30.943033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 768/1546] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:30.944957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd76675c-99ce-4f89-a350-38c942ac3ec4 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.944957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd76675c-99ce-4f89-a350-38c942ac3ec4 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.944957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd76675c-99ce-4f89-a350-38c942ac3ec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.951327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 779/1547] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:30.953031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9c47c8-5552-47d5-aa57-f3b7edb26806 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:30.965845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9c47c8-5552-47d5-aa57-f3b7edb26806 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:30.970780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae0f7bd-c2dd-4607-9265-bd2abb78fb36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:30.971594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae0f7bd-c2dd-4607-9265-bd2abb78fb36 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.971873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae0f7bd-c2dd-4607-9265-bd2abb78fb36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.972142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae0f7bd-c2dd-4607-9265-bd2abb78fb36 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.972998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae0f7bd-c2dd-4607-9265-bd2abb78fb36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:30.973460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-dd76675c-99ce-4f89-a350-38c942ac3ec4 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:30.974614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae0f7bd-c2dd-4607-9265-bd2abb78fb36 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:30.974733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dd76675c-99ce-4f89-a350-38c942ac3ec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:30.986161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 780/1548] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:30.987392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50b623-f071-4c5d-a31e-7796b0d18805 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.987794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 769/1549] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:30.987915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50b623-f071-4c5d-a31e-7796b0d18805 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.988009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c50b623-f071-4c5d-a31e-7796b0d18805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:30.988850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6389eb-30f8-47db-8d67-a69c8bb47512 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:30.991235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6389eb-30f8-47db-8d67-a69c8bb47512 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:30.991766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6389eb-30f8-47db-8d67-a69c8bb47512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.016585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4c50b623-f071-4c5d-a31e-7796b0d18805 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.017523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3e6389eb-30f8-47db-8d67-a69c8bb47512 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.017806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4c50b623-f071-4c5d-a31e-7796b0d18805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.018718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3e6389eb-30f8-47db-8d67-a69c8bb47512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.026005 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 781/1550] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.026924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 770/1551] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.027134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba498512-3e76-4a05-8942-8bb3691ace2b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.027379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba498512-3e76-4a05-8942-8bb3691ace2b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.027655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba498512-3e76-4a05-8942-8bb3691ace2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.028102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b337c-04ae-4123-a531-0c0bc17d4373 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.028383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b337c-04ae-4123-a531-0c0bc17d4373 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.028656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b337c-04ae-4123-a531-0c0bc17d4373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.047572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ba498512-3e76-4a05-8942-8bb3691ace2b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.047980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-020b337c-04ae-4123-a531-0c0bc17d4373 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.048778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ba498512-3e76-4a05-8942-8bb3691ace2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.049152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-020b337c-04ae-4123-a531-0c0bc17d4373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.056531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 782/1552] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.057146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 771/1553] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.057973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b337c-04ae-4123-a531-0c0bc17d4373 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.061333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8b38ab-6533-4792-8257-f5aecc4dfcb9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.061622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8b38ab-6533-4792-8257-f5aecc4dfcb9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.061932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8b38ab-6533-4792-8257-f5aecc4dfcb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.067725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b337c-04ae-4123-a531-0c0bc17d4373 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.072389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.072995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.073273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.073552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.074247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.075901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.081638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-db8b38ab-6533-4792-8257-f5aecc4dfcb9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.082836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-db8b38ab-6533-4792-8257-f5aecc4dfcb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.086503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 772/1554] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:31.087338 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.091306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 783/1555] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.092388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f131267-dcd4-408a-8241-ec2068870fb5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.092675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f131267-dcd4-408a-8241-ec2068870fb5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.092955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f131267-dcd4-408a-8241-ec2068870fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.097120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d01c5-8b6d-4df7-b2f1-11fa9fc7c821 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.102022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.102651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.102930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.103230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.103930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:31.104235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.105256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING py.warnings [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:10:31.105256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: warnings.warn(msg) Jul 26 03:10:31.105256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: Jul 26 03:10:31.106138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eda89e-55c4-4957-b533-50466ae668ba admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.114019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-7f131267-dcd4-408a-8241-ec2068870fb5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.115270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7f131267-dcd4-408a-8241-ec2068870fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.118400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 773/1556] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:31.119791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f56219-dbf3-48b8-8299-ae5c4c78e5e6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.120084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f56219-dbf3-48b8-8299-ae5c4c78e5e6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.120361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f56219-dbf3-48b8-8299-ae5c4c78e5e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.123343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 784/1557] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.124466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540539c6-7aa0-4a63-987a-1a1966d47d4b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.124751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540539c6-7aa0-4a63-987a-1a1966d47d4b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.125021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540539c6-7aa0-4a63-987a-1a1966d47d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.144623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-540539c6-7aa0-4a63-987a-1a1966d47d4b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.145841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-540539c6-7aa0-4a63-987a-1a1966d47d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.147944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-01f56219-dbf3-48b8-8299-ae5c4c78e5e6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.149111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-01f56219-dbf3-48b8-8299-ae5c4c78e5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.154657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 785/1558] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.155482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540539c6-7aa0-4a63-987a-1a1966d47d4b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.160765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 774/1559] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.161834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f56219-dbf3-48b8-8299-ae5c4c78e5e6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.168350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540539c6-7aa0-4a63-987a-1a1966d47d4b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.171267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f56219-dbf3-48b8-8299-ae5c4c78e5e6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.174404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.175070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.175341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.175610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.176311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:31.176613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.177142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11da37ef-7a11-4f30-8218-425878600553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.177751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da37ef-7a11-4f30-8218-425878600553 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.178028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da37ef-7a11-4f30-8218-425878600553 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.178220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.178334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da37ef-7a11-4f30-8218-425878600553 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.179008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11da37ef-7a11-4f30-8218-425878600553 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:31.179315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11da37ef-7a11-4f30-8218-425878600553 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.180934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11da37ef-7a11-4f30-8218-425878600553 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.190430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 786/1560] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:31.191236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.199942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 775/1561] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:31.199942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f45fd7-a2cc-49d0-b759-eda2c90df4e9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.199942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f45fd7-a2cc-49d0-b759-eda2c90df4e9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.199942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f45fd7-a2cc-49d0-b759-eda2c90df4e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.202085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9ebcf-1ed2-457e-a0e5-956540880637 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.207683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fedec1-35cf-4a84-9dd0-f9b28af41e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.208347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fedec1-35cf-4a84-9dd0-f9b28af41e3c admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.208626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fedec1-35cf-4a84-9dd0-f9b28af41e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.208897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fedec1-35cf-4a84-9dd0-f9b28af41e3c admin admin] PATH_INFO: `/v3/projects/a07cd1888316403eb9c37610a310b76d/users/1a1125811ba549e0ae66b26ad3949f23/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.217537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-80f45fd7-a2cc-49d0-b759-eda2c90df4e9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.218704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-80f45fd7-a2cc-49d0-b759-eda2c90df4e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.224196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fedec1-35cf-4a84-9dd0-f9b28af41e3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a07cd1888316403eb9c37610a310b76d, user_id=1a1125811ba549e0ae66b26ad3949f23, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.225229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-05fedec1-35cf-4a84-9dd0-f9b28af41e3c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:10:31.225229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:10:31.225229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:10:31.227217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fedec1-35cf-4a84-9dd0-f9b28af41e3c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.227568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 776/1562] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.228333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f45fd7-a2cc-49d0-b759-eda2c90df4e9 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.238104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f45fd7-a2cc-49d0-b759-eda2c90df4e9 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.242525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 787/1563] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:31 2022] PUT /identity/v3/projects/a07cd1888316403eb9c37610a310b76d/users/1a1125811ba549e0ae66b26ad3949f23/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:31.243821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a91fd-0763-45fe-bd62-d8cc4edeea41 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.244107 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a91fd-0763-45fe-bd62-d8cc4edeea41 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.244382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a91fd-0763-45fe-bd62-d8cc4edeea41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.263204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-650a91fd-0763-45fe-bd62-d8cc4edeea41 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.264451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-650a91fd-0763-45fe-bd62-d8cc4edeea41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.282797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:31.296447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.297074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.297351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.297623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.298288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.300226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.304005 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:31.325650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 788/1564] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.326752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650a91fd-0763-45fe-bd62-d8cc4edeea41 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.329740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 777/1565] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/projects => generated 371 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.330600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.338393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650a91fd-0763-45fe-bd62-d8cc4edeea41 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.393676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.394284 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.394498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a2742e-386f-4241-ac87-36f0563801fb admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.394644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.394824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.395626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:31.395938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.397550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.399004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.399651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.399931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.400204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.400900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:31.401206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.402805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.408800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 789/1566] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:31.409649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.420584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 778/1567] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:31.420836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b22605b-b9b0-417d-b253-594a4a315ee1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.421409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.425706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.426309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.426588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.426859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.427550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.428559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:10:31.428559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:10:31.428559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:10:31.430008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.432518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fb43d2-1fc2-4f52-8d35-3b3e7813e58e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.437835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6eaf6e-e296-4a9e-80e3-a6c36a0a38a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.438448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6eaf6e-e296-4a9e-80e3-a6c36a0a38a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.438729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6eaf6e-e296-4a9e-80e3-a6c36a0a38a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.438997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6eaf6e-e296-4a9e-80e3-a6c36a0a38a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.439692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6eaf6e-e296-4a9e-80e3-a6c36a0a38a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.441579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6eaf6e-e296-4a9e-80e3-a6c36a0a38a5 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.458811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 790/1568] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.460002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.470107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 779/1569] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.471728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ad5f0-be63-41ce-8d1e-dfd4a7eecd93 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.472182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ad5f0-be63-41ce-8d1e-dfd4a7eecd93 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.472459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6ad5f0-be63-41ce-8d1e-dfd4a7eecd93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.490994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-5b6ad5f0-be63-41ce-8d1e-dfd4a7eecd93 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:31.492144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5b6ad5f0-be63-41ce-8d1e-dfd4a7eecd93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:31.523748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a685b904-7b35-4226-baa1-ab20739039a2 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.550923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 780/1570] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:31.551785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6ad5f0-be63-41ce-8d1e-dfd4a7eecd93 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.562467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6ad5f0-be63-41ce-8d1e-dfd4a7eecd93 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.567868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.568489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.568777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.569053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.569727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.570768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING py.warnings [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:10:31.570768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: warnings.warn(msg) Jul 26 03:10:31.570768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: Jul 26 03:10:31.572030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.582601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.583338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.583638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.583813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.584509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:31.584800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.586404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.589019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:31.596065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 781/1571] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/users => generated 286 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.596888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.609779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 791/1572] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:31.609779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.613019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec68ac-970d-4bdf-9c13-97bee48e3249 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.613996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d730a8-9c3c-4bb4-a015-81f665a4ec2c admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.617578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.618301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.618433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.618626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.618852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.618986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.619261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.619534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.619670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.620229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:31.620530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.621458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.622118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.634310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 782/1573] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:31.635888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.647373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c680ead1-78c6-44e9-9f5c-da5b4c6749f5 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.651910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.652071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 792/1574] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.652516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.652796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.652912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.653089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.653701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.655944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.667722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:31.687738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 783/1575] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.687738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.717718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44758755-ae65-4f0b-80f0-b26c9b0b2592 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.747385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b31b913-57f4-4193-8594-1e619e909d49 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.752038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.752665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.752950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.753253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.753932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.756011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.774592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.775262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.775544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.775816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.776477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.777364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 784/1576] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/users => generated 286 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.778278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.778413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.800797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 793/1577] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/users => generated 288 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.801667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.811669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6339dd55-34aa-46b7-a345-2bdeb7cc289c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.816515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.817154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.817437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.817717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.818411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.820205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.830696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 794/1578] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:31.831602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.840295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7798dacb-ac90-4cd4-bec9-aa9e256f5a5d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.841001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc932445-6212-4ea1-bf93-4c7e769a96c8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.844223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.844908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.845312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.845589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.845718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.846315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.846591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.846857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.847540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.849460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.855029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.856505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.866247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 785/1579] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:31.867135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.876627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7b0c5a-b14d-4875-8393-8dd3f9c45c9b service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.882175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.882767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.883083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.883351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:31.884019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:31.886289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:31.887866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 795/1580] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/projects => generated 377 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.888904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.918638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 786/1581] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/projects => generated 387 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:31.919559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:31.937172 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:31.965126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab919c8-657c-4e22-99d0-a4e1bfe805cc admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.989850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87d81c3-046b-450d-805c-37ea69d328a8 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:31.998646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:31.999367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:31.999661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:31.999911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.000619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.002609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.013652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:32.028964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 787/1582] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/users => generated 287 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:32.031143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.038454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.039267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.039554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.039830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.040554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.041475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a034d0-07c9-4110-a1bd-29a9aa4ba6b6 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.042564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.046641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.047359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.047705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.048270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.048957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.050835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.067056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 796/1583] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:31 2022] POST /identity/v3/users => generated 292 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:32.068131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.073839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 788/1584] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.074701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.083867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd591d3-a776-4197-a5a7-9324d6b5aef3 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.087163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c8e690-1360-44a7-ac24-45a99676feda admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.088519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.089145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.089422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.089727 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.090431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.092121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.092239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.092889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.093354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.093699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.094465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.096300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.106261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 797/1585] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.106261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.113267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 789/1586] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.113267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.116346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8441fc-98e3-4dc0-846b-0f34aa5f2191 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.121186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.121813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.122095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.122372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.123079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.124317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba652d24-cde5-40d4-857d-8211ddabb1f8 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.125051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.128856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.129668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.130013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.130446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.131236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.133512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.148319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 798/1587] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:32 2022] POST /identity/v3/users => generated 289 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:32.149363 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.162856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 790/1588] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:32 2022] POST /identity/v3/users => generated 294 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:32.163403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c17f0a1-5845-4e17-bf20-93f3dc5470a9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.163527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.167884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.168824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.169164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.169509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.170361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.172146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.172251 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f76b1c6-24da-48f2-b854-034c1b5b629d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.176726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.177340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.177619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.177888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.178590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.180246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.184085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 799/1589] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.185377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.192638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 791/1590] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.193599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.202718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e32e5-f612-47ae-80ee-45ba847430c4 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.207480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0bcd7a-7974-4522-a4ca-8adc2fe3e396 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.208567 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.209395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.209675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.209944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.210646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.212293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.213410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.214274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.214753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.215382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.216284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.218344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.227603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 800/1591] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.227603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.236966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 792/1592] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.237840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.246556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56ff894-9ca0-495c-81c5-e4db01dfaacd admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.249944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b6e55-0995-44d0-ad29-53710d1de775 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.252385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.253365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.253808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.254085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] PATH_INFO: `/v3/projects/82b81d9e135a42fa921d59ced0ead98f/users/2d08f38d9a9445ba9be4a83efa668cab/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.255281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.256083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.256565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.256840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.257610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.259344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.273373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 793/1593] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.274401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.275947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82b81d9e135a42fa921d59ced0ead98f, user_id=2d08f38d9a9445ba9be4a83efa668cab, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.279242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.285270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f4d500-4bba-46a6-80ee-b074c28e8e83 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.293100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.293825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.294213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.294623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.295641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.297665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.315193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 801/1594] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:32 2022] PUT /identity/v3/projects/82b81d9e135a42fa921d59ced0ead98f/users/2d08f38d9a9445ba9be4a83efa668cab/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:32.315193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.321427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 794/1595] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.321427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.334398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a00e277-1ec6-4593-a5d7-6af2f47373f4 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.386083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394c8d9d-bdf6-4f3f-a8a8-2b67565e8fb5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.391471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.392097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.392379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.392647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.393351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.394999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.396348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:32.399385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.400227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.400980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.401261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.402198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.404198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.410862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 802/1596] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.411934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.415438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 795/1597] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.416149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.422976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7934fa6e-da22-45aa-8e4c-7d12acb3aa62 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.429039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.429676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b850504-3846-4837-8ccf-a3d05a0fe8ae admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.429825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.430201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.430547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] PATH_INFO: `/v3/projects/742e204cb5d249459352af9a37abfeb4/users/505dc808299b4aed943c1640f145393b/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.436937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a15097-c1d0-44b3-8206-db69bf04ed49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.437906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a15097-c1d0-44b3-8206-db69bf04ed49 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.438332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a15097-c1d0-44b3-8206-db69bf04ed49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.438765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a15097-c1d0-44b3-8206-db69bf04ed49 admin admin] PATH_INFO: `/v3/projects/6daba0980548418ea98d4d0cf42ef92f/users/fbdac72f58df4335b3790c5fb1609f5b/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.446804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=742e204cb5d249459352af9a37abfeb4, user_id=505dc808299b4aed943c1640f145393b, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.449724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.459763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a15097-c1d0-44b3-8206-db69bf04ed49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6daba0980548418ea98d4d0cf42ef92f, user_id=fbdac72f58df4335b3790c5fb1609f5b, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.463918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a15097-c1d0-44b3-8206-db69bf04ed49 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.467837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 803/1598] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:32 2022] PUT /identity/v3/projects/742e204cb5d249459352af9a37abfeb4/users/505dc808299b4aed943c1640f145393b/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:32.475380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.481436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 796/1599] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:32 2022] PUT /identity/v3/projects/6daba0980548418ea98d4d0cf42ef92f/users/fbdac72f58df4335b3790c5fb1609f5b/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:32.482827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e07dce-4ef1-4512-8562-8ecb4d4a7741 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.483162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e07dce-4ef1-4512-8562-8ecb4d4a7741 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.483441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e07dce-4ef1-4512-8562-8ecb4d4a7741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.515341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c4e07dce-4ef1-4512-8562-8ecb4d4a7741 None None] MFA Rules not processed for user `2d08f38d9a9445ba9be4a83efa668cab`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:32.516580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c4e07dce-4ef1-4512-8562-8ecb4d4a7741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:32.553156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:32.561756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25b2324-974b-4049-b59b-eb44af4465bd admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.603432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 797/1600] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:32 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:32.604384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e07dce-4ef1-4512-8562-8ecb4d4a7741 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.642176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.643165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.643458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.643737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.644451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.646698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.659808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 804/1601] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.663961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.674400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501d01d5-9c46-47b8-83e4-0fcef5e67f2d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.679677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b546e5-3b9a-43d6-bc68-4a69cfe5ef7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.681409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b546e5-3b9a-43d6-bc68-4a69cfe5ef7e admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.681922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b546e5-3b9a-43d6-bc68-4a69cfe5ef7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.682183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b546e5-3b9a-43d6-bc68-4a69cfe5ef7e admin admin] PATH_INFO: `/v3/projects/bc9749de1e594aad9567e9da7e404291/users/504dbd85957345e699eba48ddab99ed2/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.689011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e07dce-4ef1-4512-8562-8ecb4d4a7741 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.695394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.696044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.696305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.696729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.698155 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.700152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.702709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b546e5-3b9a-43d6-bc68-4a69cfe5ef7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc9749de1e594aad9567e9da7e404291, user_id=504dbd85957345e699eba48ddab99ed2, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.706550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b546e5-3b9a-43d6-bc68-4a69cfe5ef7e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.713276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 798/1602] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:32 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:32.714180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.728153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 805/1603] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:32 2022] PUT /identity/v3/projects/bc9749de1e594aad9567e9da7e404291/users/504dbd85957345e699eba48ddab99ed2/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:32.728386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fe8a3-6311-46fa-9e8b-02f4ff63e7e9 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:32.729289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ec57b-b47b-46ce-b9e8-3a47c12baba7 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.729574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ec57b-b47b-46ce-b9e8-3a47c12baba7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.729851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ec57b-b47b-46ce-b9e8-3a47c12baba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.782384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b84ec57b-b47b-46ce-b9e8-3a47c12baba7 None None] MFA Rules not processed for user `505dc808299b4aed943c1640f145393b`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:32.783591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b84ec57b-b47b-46ce-b9e8-3a47c12baba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:32.825764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:32.827377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.827689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.827919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] PATH_INFO: `/v3/projects/9d8d482c37d849cf8bdf035e110e0f34/users/9d3fd2b3c6cd47019e15fbf959d47381/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.841438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:32.856283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d8d482c37d849cf8bdf035e110e0f34, user_id=9d3fd2b3c6cd47019e15fbf959d47381, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:32.860966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:32.883300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 806/1604] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:32 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:32.883300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea275d-c542-4421-9a1d-729e3bf154d7 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:32.883300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea275d-c542-4421-9a1d-729e3bf154d7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:32.883300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea275d-c542-4421-9a1d-729e3bf154d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:32.883719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 799/1605] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:32 2022] PUT /identity/v3/projects/9d8d482c37d849cf8bdf035e110e0f34/users/9d3fd2b3c6cd47019e15fbf959d47381/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:32.891416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:32.906939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-59ea275d-c542-4421-9a1d-729e3bf154d7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:32.908232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-59ea275d-c542-4421-9a1d-729e3bf154d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:32.981127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:33.007363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd87d5-fd72-48c9-af46-b57575cbc5c7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.022764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 807/1606] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:33.026815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ea275d-c542-4421-9a1d-729e3bf154d7 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.027140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8c8f9a-05aa-4d3a-a863-212f0b20da5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.027907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c8f9a-05aa-4d3a-a863-212f0b20da5f admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.028192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c8f9a-05aa-4d3a-a863-212f0b20da5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.028470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8c8f9a-05aa-4d3a-a863-212f0b20da5f admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.029164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8c8f9a-05aa-4d3a-a863-212f0b20da5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.033910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8c8f9a-05aa-4d3a-a863-212f0b20da5f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.037746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ea275d-c542-4421-9a1d-729e3bf154d7 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.044938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.046162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.046162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.046366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] PATH_INFO: `/v3/projects/e5b0edb065564485bebea31ef6f20b7e/users/adcae81584cc447db8d2628ff6e1bf5b/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.064088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5b0edb065564485bebea31ef6f20b7e, user_id=adcae81584cc447db8d2628ff6e1bf5b, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.067470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.087058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 808/1607] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:33 2022] PUT /identity/v3/projects/e5b0edb065564485bebea31ef6f20b7e/users/adcae81584cc447db8d2628ff6e1bf5b/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:33.087276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.092199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 800/1608] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:32 2022] POST /identity/v3/projects => generated 369 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:33.094713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918880a-f7f9-4289-9e8d-c8ede52b363c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.095696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918880a-f7f9-4289-9e8d-c8ede52b363c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.095987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918880a-f7f9-4289-9e8d-c8ede52b363c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.142922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7918880a-f7f9-4289-9e8d-c8ede52b363c None None] MFA Rules not processed for user `504dbd85957345e699eba48ddab99ed2`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:33.144285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7918880a-f7f9-4289-9e8d-c8ede52b363c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:33.196433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adf4c77-ee14-46c2-a3b1-20ebb013a3f4 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.236277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 801/1609] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:33.237830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a7bf1-4470-4546-bc71-c1dd0efbdbcf None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.238305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a7bf1-4470-4546-bc71-c1dd0efbdbcf None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.238973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541a7bf1-4470-4546-bc71-c1dd0efbdbcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.276217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547248b6-6248-418c-9903-747387ce90ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.277091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547248b6-6248-418c-9903-747387ce90ba admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.277433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547248b6-6248-418c-9903-747387ce90ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.277890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547248b6-6248-418c-9903-747387ce90ba admin admin] PATH_INFO: `/v3/projects/ab741435041940cc9c7f02b27120bd90/users/037c9f8229a44de198ebc3752f7a1966/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.290170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:33.301301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547248b6-6248-418c-9903-747387ce90ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab741435041940cc9c7f02b27120bd90, user_id=037c9f8229a44de198ebc3752f7a1966, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.305446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547248b6-6248-418c-9903-747387ce90ba admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.305786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-541a7bf1-4470-4546-bc71-c1dd0efbdbcf None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:33.311650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-541a7bf1-4470-4546-bc71-c1dd0efbdbcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:33.324931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 809/1610] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:33 2022] PUT /identity/v3/projects/ab741435041940cc9c7f02b27120bd90/users/037c9f8229a44de198ebc3752f7a1966/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:33.326889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41e4dc-e79e-4937-84ca-bc32b5dd58dd None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.327199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41e4dc-e79e-4937-84ca-bc32b5dd58dd None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.327472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41e4dc-e79e-4937-84ca-bc32b5dd58dd None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.329640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 810/1611] 10.222.0.42 () {54 vars in 962 bytes} [Tue Jul 26 03:10:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:10:33.330377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a41e4dc-e79e-4937-84ca-bc32b5dd58dd None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.332339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2a41e4dc-e79e-4937-84ca-bc32b5dd58dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:33.332747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 802/1612] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:33.334497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5aa68-fe6f-4082-afd6-20b417e63c3c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.335132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5aa68-fe6f-4082-afd6-20b417e63c3c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.335637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb5aa68-fe6f-4082-afd6-20b417e63c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.369551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7fb5aa68-fe6f-4082-afd6-20b417e63c3c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:33.371734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7fb5aa68-fe6f-4082-afd6-20b417e63c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:33.439668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a41e4dc-e79e-4937-84ca-bc32b5dd58dd None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.444423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f4332e-c48c-4bc0-8af0-cd573afed4ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.445445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4332e-c48c-4bc0-8af0-cd573afed4ab service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.445821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4332e-c48c-4bc0-8af0-cd573afed4ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.446184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4332e-c48c-4bc0-8af0-cd573afed4ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.471154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 803/1613] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:33.472118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5aa68-fe6f-4082-afd6-20b417e63c3c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.486134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb5aa68-fe6f-4082-afd6-20b417e63c3c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.496287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1359976a-01c4-4bba-b08e-bb5e14b4f359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.496952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359976a-01c4-4bba-b08e-bb5e14b4f359 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.497240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359976a-01c4-4bba-b08e-bb5e14b4f359 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.497502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359976a-01c4-4bba-b08e-bb5e14b4f359 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.498241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1359976a-01c4-4bba-b08e-bb5e14b4f359 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.500249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1359976a-01c4-4bba-b08e-bb5e14b4f359 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.537057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f4332e-c48c-4bc0-8af0-cd573afed4ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.538721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f4332e-c48c-4bc0-8af0-cd573afed4ab service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.539447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 804/1614] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:33 2022] POST /identity/v3/users => generated 291 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:33.541124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628e9a62-0c8e-4cc6-8b02-e20f3186c041 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.541694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628e9a62-0c8e-4cc6-8b02-e20f3186c041 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.542392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628e9a62-0c8e-4cc6-8b02-e20f3186c041 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.549790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 805/1615] 10.222.0.42 () {54 vars in 962 bytes} [Tue Jul 26 03:10:33 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:10:33.551244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1df6ea-d81d-4385-a65d-779c18da267e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.551510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 811/1616] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:33.553072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c32a95-a14d-4479-be18-568d4b1c7d25 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.553449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c32a95-a14d-4479-be18-568d4b1c7d25 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.553935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c32a95-a14d-4479-be18-568d4b1c7d25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.566921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1df6ea-d81d-4385-a65d-779c18da267e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.567525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1df6ea-d81d-4385-a65d-779c18da267e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.596514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-0e1df6ea-d81d-4385-a65d-779c18da267e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:33.599379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0e1df6ea-d81d-4385-a65d-779c18da267e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:33.613007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 806/1617] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:33.613007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1df6ea-d81d-4385-a65d-779c18da267e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.613388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a1c32a95-a14d-4479-be18-568d4b1c7d25 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:33.613388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a1c32a95-a14d-4479-be18-568d4b1c7d25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:33.627106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1df6ea-d81d-4385-a65d-779c18da267e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.632679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.633966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.634423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.634842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.691518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.691518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.691518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 807/1618] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:33.691518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.691518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2307043a-3d88-4e2a-9ff9-0f11783e0614 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.704987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.705644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.705923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.706354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.740947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.742906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.756048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 808/1619] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:33.757070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.761360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 812/1620] 10.222.0.42 () {58 vars in 1059 bytes} [Tue Jul 26 03:10:33 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:33.763986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c33883-83e1-4a51-970e-f4dd3b8c1f8c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.764948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c33883-83e1-4a51-970e-f4dd3b8c1f8c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.765324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c33883-83e1-4a51-970e-f4dd3b8c1f8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.771595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354ff33-e67c-4cf5-b26f-aa449a776276 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.781600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.782545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.782895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.783271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.784055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.785846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.788166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:33.807334 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-32c33883-83e1-4a51-970e-f4dd3b8c1f8c None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:33.809340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-32c33883-83e1-4a51-970e-f4dd3b8c1f8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:33.812259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 809/1621] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:33.813191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.828519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 813/1622] 10.222.0.42 () {58 vars in 1059 bytes} [Tue Jul 26 03:10:33 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:33.829787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c33883-83e1-4a51-970e-f4dd3b8c1f8c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.841075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec5b84f-7de1-419b-93f5-6d2a29319798 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.843853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c33883-83e1-4a51-970e-f4dd3b8c1f8c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.845820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.846851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.846997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.847357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.850206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.850912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.851499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.851799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.862159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.864457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.865690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:33.884382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 810/1623] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:33.890407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.914701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3c6e6-7e12-4cf7-a77e-c21219d9dda0 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.921774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:33.925428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:33.927689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:33.927995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] PATH_INFO: `/v3/projects/8757ee77a1884f24a4bddbd2f86ff27d/users/774315b3525543c196646cfbc81df8f0/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:33.943391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.947046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.953947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8757ee77a1884f24a4bddbd2f86ff27d, user_id=774315b3525543c196646cfbc81df8f0, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:33.958891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:33.961699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 814/1624] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:33.963406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:33.977196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d356f35-59e2-41f2-95b6-68c351c0aeda service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:33.986101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 811/1625] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:33 2022] PUT /identity/v3/projects/8757ee77a1884f24a4bddbd2f86ff27d/users/774315b3525543c196646cfbc81df8f0/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:33.992128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:34.075932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114d4b90-d561-42c2-b5bd-f6e21827353b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:34.076689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114d4b90-d561-42c2-b5bd-f6e21827353b service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.077204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114d4b90-d561-42c2-b5bd-f6e21827353b service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.077822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114d4b90-d561-42c2-b5bd-f6e21827353b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.097391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181bc1bf-2194-4a00-bb63-b876bd711030 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:34.104815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:34.106247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.106901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.107479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.108514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:34.110912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:34.130079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 812/1626] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/roles => generated 1420 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:34.132017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:34.151594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c0b0bb-4694-4edd-a42e-ca0c442233c6 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:34.152591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114d4b90-d561-42c2-b5bd-f6e21827353b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:34.154214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114d4b90-d561-42c2-b5bd-f6e21827353b service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:34.157755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6564d157-fd25-4871-bb8d-3b90e6289b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:34.158447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564d157-fd25-4871-bb8d-3b90e6289b73 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.158725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564d157-fd25-4871-bb8d-3b90e6289b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.159441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564d157-fd25-4871-bb8d-3b90e6289b73 admin admin] PATH_INFO: `/v3/projects/8757ee77a1884f24a4bddbd2f86ff27d/users/774315b3525543c196646cfbc81df8f0/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.161435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6564d157-fd25-4871-bb8d-3b90e6289b73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8757ee77a1884f24a4bddbd2f86ff27d, user_id=774315b3525543c196646cfbc81df8f0, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:34.165220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6564d157-fd25-4871-bb8d-3b90e6289b73 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:34.165467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 815/1627] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:33 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:34.189410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 813/1628] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:34 2022] PUT /identity/v3/projects/8757ee77a1884f24a4bddbd2f86ff27d/users/774315b3525543c196646cfbc81df8f0/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:34.190166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939ab54-5ce1-481f-969b-3f12e5fe385e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.190462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939ab54-5ce1-481f-969b-3f12e5fe385e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.190747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939ab54-5ce1-481f-969b-3f12e5fe385e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.254304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c939ab54-5ce1-481f-969b-3f12e5fe385e None None] MFA Rules not processed for user `774315b3525543c196646cfbc81df8f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:34.255620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c939ab54-5ce1-481f-969b-3f12e5fe385e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:34.325043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:34.365922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e1de73-2046-46a7-8bdd-6aa95a7f33fc None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.366267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e1de73-2046-46a7-8bdd-6aa95a7f33fc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.366482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e1de73-2046-46a7-8bdd-6aa95a7f33fc None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.368442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 814/1629] 10.222.0.42 () {54 vars in 912 bytes} [Tue Jul 26 03:10:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:10:34.379308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d25371a-351b-4837-a5b3-da25e6198020 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.379635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d25371a-351b-4837-a5b3-da25e6198020 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.379896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d25371a-351b-4837-a5b3-da25e6198020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.407322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 816/1630] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:34 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:34.419895 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939ab54-5ce1-481f-969b-3f12e5fe385e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:34.431040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3d25371a-351b-4837-a5b3-da25e6198020 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:34.433219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3d25371a-351b-4837-a5b3-da25e6198020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:34.547508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939ab54-5ce1-481f-969b-3f12e5fe385e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:34.560892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 815/1631] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:10:34 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:34.584766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d25371a-351b-4837-a5b3-da25e6198020 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:34.586077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2080b07-0274-4fed-a766-0117fed064ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:34.586749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2080b07-0274-4fed-a766-0117fed064ba service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.587066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2080b07-0274-4fed-a766-0117fed064ba service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.587354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2080b07-0274-4fed-a766-0117fed064ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.602445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d25371a-351b-4837-a5b3-da25e6198020 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:34.611198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:34.611951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.612344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.612690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.626523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2080b07-0274-4fed-a766-0117fed064ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:34.629249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2080b07-0274-4fed-a766-0117fed064ba service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:34.639361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:34.640926 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:34.649773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 817/1632] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:34 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:10:34.651604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5bd79-4d83-4f94-ae71-de5bd0bc94e6 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.651940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5bd79-4d83-4f94-ae71-de5bd0bc94e6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.652217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5bd79-4d83-4f94-ae71-de5bd0bc94e6 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.654263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 818/1633] 10.222.0.42 () {54 vars in 912 bytes} [Tue Jul 26 03:10:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:10:34.661385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41febdc-8ffb-48b9-96d4-da192c3f3f84 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.661853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41febdc-8ffb-48b9-96d4-da192c3f3f84 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.662151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41febdc-8ffb-48b9-96d4-da192c3f3f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.676509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 816/1634] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:34 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:10:34.708483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d41febdc-8ffb-48b9-96d4-da192c3f3f84 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:34.710725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d41febdc-8ffb-48b9-96d4-da192c3f3f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:34.752461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 819/1635] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:10:34 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:34.787064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:34.797097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b7e939-5a8d-4955-8f10-f4824bab648e service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:34.805623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8863ec-b974-4e30-8c00-ce90696e5971 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:34.806651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8863ec-b974-4e30-8c00-ce90696e5971 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:34.806651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8863ec-b974-4e30-8c00-ce90696e5971 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:34.806872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8863ec-b974-4e30-8c00-ce90696e5971 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:34.829641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8863ec-b974-4e30-8c00-ce90696e5971 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:34.831172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8863ec-b974-4e30-8c00-ce90696e5971 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:34.860885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 817/1636] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:34 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:35.368021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41febdc-8ffb-48b9-96d4-da192c3f3f84 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:35.465169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41febdc-8ffb-48b9-96d4-da192c3f3f84 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:35.474984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:35.475356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:35.475801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:35.476080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] PATH_INFO: `/v3/users/504dbd85957345e699eba48ddab99ed2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:35.477278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=504dbd85957345e699eba48ddab99ed2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:35.478320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: WARNING py.warnings [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:10:35.478320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: warnings.warn(msg) Jul 26 03:10:35.478320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: Jul 26 03:10:35.480623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:35.699052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '504dbd85957345e699eba48ddab99ed2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:35.789557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '504dbd85957345e699eba48ddab99ed2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:35.797613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:35.810992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '504dbd85957345e699eba48ddab99ed2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:35.831402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '504dbd85957345e699eba48ddab99ed2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:35.832703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '504dbd85957345e699eba48ddab99ed2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:35.848756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 820/1637] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:10:35 2022] DELETE /identity/v3/users/504dbd85957345e699eba48ddab99ed2 => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:35.853352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:35.853882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:35.853882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:35.896693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:35.914833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:35.918678 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:36.035570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:36.037974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:36.071668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 818/1638] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:36.156867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:36.159107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:36.161779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e9305-f421-414c-824a-c2298b6fbe35 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:36.168545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:36.170065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:36.170597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:36.171271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] PATH_INFO: `/v3/users/2d08f38d9a9445ba9be4a83efa668cab` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:36.173118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d08f38d9a9445ba9be4a83efa668cab)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:36.175635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:36.286604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d08f38d9a9445ba9be4a83efa668cab'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:36.316060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44078bca-50de-45b0-b427-b2dfe0db7a49 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:36.321055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d08f38d9a9445ba9be4a83efa668cab'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:36.321683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c7703-95e8-41b6-822f-1d49445c1b56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:36.322332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c7703-95e8-41b6-822f-1d49445c1b56 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:36.322616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c7703-95e8-41b6-822f-1d49445c1b56 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:36.322964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c7703-95e8-41b6-822f-1d49445c1b56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:36.338751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d08f38d9a9445ba9be4a83efa668cab'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:36.343360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c7703-95e8-41b6-822f-1d49445c1b56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:36.346502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c7703-95e8-41b6-822f-1d49445c1b56 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:36.361475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d08f38d9a9445ba9be4a83efa668cab'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:36.362568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d08f38d9a9445ba9be4a83efa668cab'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:36.368353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 819/1639] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:10:36.379571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 821/1640] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:10:36 2022] DELETE /identity/v3/users/2d08f38d9a9445ba9be4a83efa668cab => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:36.384347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:36.385154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:36.386772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:36.417398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:36.418984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:36.519419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 820/1641] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:10:36.581377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:36.583201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:36.763420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de850278-8098-495b-99a3-7f1a82034f39 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:36.781090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:36.781878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:36.782161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:36.782435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:36.814468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:36.816905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:36.834246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 822/1642] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:38.346395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:38.348061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:38.366168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adef7542-4a3c-4ba9-ae28-e84e772cf19a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:38.389393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:38.390553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:38.390553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:38.392519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] PATH_INFO: `/v3/projects/82b81d9e135a42fa921d59ced0ead98f` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:38.398224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82b81d9e135a42fa921d59ced0ead98f)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:38.400084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: WARNING py.warnings [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 26 03:10:38.400084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: warnings.warn(msg) Jul 26 03:10:38.400084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: Jul 26 03:10:38.402589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:38.524022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:38.526106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:38.538744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:38.549534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82b81d9e135a42fa921d59ced0ead98f', 'request_id': 'req-75792377-4851-438e-8165-1fc23facd525'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:38.559583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5b1d1d-1cd8-43a2-9dd0-4897b18809c1 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:38.567203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82b81d9e135a42fa921d59ced0ead98f', 'request_id': 'req-75792377-4851-438e-8165-1fc23facd525'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:38.573560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:38.587869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82b81d9e135a42fa921d59ced0ead98f', 'request_id': 'req-75792377-4851-438e-8165-1fc23facd525'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:38.636099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] The token cache is being invalidate because project 82b81d9e135a42fa921d59ced0ead98f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:10:38.636414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:38.642700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:38.644857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 821/1643] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:10:38 2022] DELETE /identity/v3/projects/82b81d9e135a42fa921d59ced0ead98f => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:38.648630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:38.650901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:38.820321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:38.822659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:38.824491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:38.825352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] PATH_INFO: `/v3/users/505dc808299b4aed943c1640f145393b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:38.828530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] RBAC: Authorizing `identity:delete_user(user_id=505dc808299b4aed943c1640f145393b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:38.849779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75792377-4851-438e-8165-1fc23facd525 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:38.860657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:38.864648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:38.865345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:38.865628 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:38.865907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] PATH_INFO: `/v3/users/774315b3525543c196646cfbc81df8f0` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:38.869341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] RBAC: Authorizing `identity:delete_user(user_id=774315b3525543c196646cfbc81df8f0)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:38.871516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:38.975551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '774315b3525543c196646cfbc81df8f0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:38.991836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '774315b3525543c196646cfbc81df8f0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:38.994263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '505dc808299b4aed943c1640f145393b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.017425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '505dc808299b4aed943c1640f145393b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.021761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '774315b3525543c196646cfbc81df8f0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.029929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '505dc808299b4aed943c1640f145393b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.044644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '774315b3525543c196646cfbc81df8f0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.045595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a4cb60ed-9798-45c6-b2c5-80d956aaa550 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '774315b3525543c196646cfbc81df8f0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.047978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '505dc808299b4aed943c1640f145393b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.049366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '505dc808299b4aed943c1640f145393b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.058566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 822/1644] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:10:38 2022] DELETE /identity/v3/users/774315b3525543c196646cfbc81df8f0 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:39.062800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 823/1645] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:10:38 2022] DELETE /identity/v3/users/505dc808299b4aed943c1640f145393b => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:10:39.067741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:39.068528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:39.068914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:39.103588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:39.104854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:39.183835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:39.210330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 823/1646] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:39.240720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:39.242824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:39.358133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a2237-1124-4261-bafd-8beef7d7a21a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:39.365685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716565f6-a9d8-427f-9928-8773ab126c21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:39.366728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716565f6-a9d8-427f-9928-8773ab126c21 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:39.366830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716565f6-a9d8-427f-9928-8773ab126c21 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:39.367091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716565f6-a9d8-427f-9928-8773ab126c21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:39.381951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716565f6-a9d8-427f-9928-8773ab126c21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:39.384489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716565f6-a9d8-427f-9928-8773ab126c21 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:39.398987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 824/1647] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:39.555064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:39.558442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:39.581815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004ce509-2bdb-4e7d-aa94-b25ae3f716bb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:39.596726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:39.600955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:39.602795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:39.603361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] PATH_INFO: `/v3/projects/bc9749de1e594aad9567e9da7e404291` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:39.609951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc9749de1e594aad9567e9da7e404291)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:39.619160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:39.761424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc9749de1e594aad9567e9da7e404291', 'request_id': 'req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.789510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc9749de1e594aad9567e9da7e404291', 'request_id': 'req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.807426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc9749de1e594aad9567e9da7e404291', 'request_id': 'req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.830279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] The token cache is being invalidate because project bc9749de1e594aad9567e9da7e404291 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:10:39.830684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.831917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5369c2c9-6bf2-4886-80df-d5fe4a5a42b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:39.836927 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 824/1648] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:10:39 2022] DELETE /identity/v3/projects/bc9749de1e594aad9567e9da7e404291 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:39.838782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f60ab5-9d30-4c05-85ff-8378632f37b8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:39.839296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f60ab5-9d30-4c05-85ff-8378632f37b8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:39.839364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f60ab5-9d30-4c05-85ff-8378632f37b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:39.905035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-66f60ab5-9d30-4c05-85ff-8378632f37b8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:39.906504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-66f60ab5-9d30-4c05-85ff-8378632f37b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:39.953294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:40.061701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 825/1649] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:40.070344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e36e765-ced2-47bb-b1e0-68faf41a3714 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.070641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e36e765-ced2-47bb-b1e0-68faf41a3714 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.071129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e36e765-ced2-47bb-b1e0-68faf41a3714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.150930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9e36e765-ced2-47bb-b1e0-68faf41a3714 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:40.152269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9e36e765-ced2-47bb-b1e0-68faf41a3714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:40.161183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 825/1650] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:40.165939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f60ab5-9d30-4c05-85ff-8378632f37b8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:40.186476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f60ab5-9d30-4c05-85ff-8378632f37b8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:40.201505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:40.203950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.204606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.204886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.205636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:40.205947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:40.219448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:40.262718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 826/1651] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:40.264404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e36e765-ced2-47bb-b1e0-68faf41a3714 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:40.283255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e36e765-ced2-47bb-b1e0-68faf41a3714 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:40.296304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:40.296304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.297340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.297340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.297726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:40.300892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:40.436616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 826/1652] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:40 2022] POST /identity/v3/projects => generated 359 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:40.440593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:40.456457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:40.458296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684bd48d-3ad4-48b2-a893-75cb8d1bd478 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:40.462027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:40.556236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:40.560723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.561109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.561403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.562122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:40.565810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:40.616909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 827/1653] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:40 2022] POST /identity/v3/users => generated 280 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:40.618099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:40.619742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91acb69-6b52-41e6-b4cb-90126a93dc1d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:40.625489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:40.626386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.626769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.627149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] PATH_INFO: `/v3/projects/742e204cb5d249459352af9a37abfeb4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.629451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] RBAC: Authorizing `identity:delete_project(project_id=742e204cb5d249459352af9a37abfeb4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:40.631702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:40.634416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90a043d-b038-4deb-90d5-96b9d0342d2b admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:40.649393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:40.650035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.650320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.650598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.651553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:40.654816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:40.677422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 828/1654] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:10:40.678455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:40.689930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230f4ebb-4f56-4f99-87d6-ff51c64aefaa admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:40.696627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:40.697393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.697702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.698045 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.699024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:40.700608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:40.710095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '742e204cb5d249459352af9a37abfeb4', 'request_id': 'req-a3449974-ff32-4f26-9a71-0fb76111940f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:40.712400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:40.717740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 829/1655] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:40.719106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:40.723836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '742e204cb5d249459352af9a37abfeb4', 'request_id': 'req-a3449974-ff32-4f26-9a71-0fb76111940f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:40.746453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afea5cc-fed9-4ff3-a3d1-75611b480c1c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:40.748470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '742e204cb5d249459352af9a37abfeb4', 'request_id': 'req-a3449974-ff32-4f26-9a71-0fb76111940f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:40.770009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] The token cache is being invalidate because project 742e204cb5d249459352af9a37abfeb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:10:40.770653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:40.772336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a3449974-ff32-4f26-9a71-0fb76111940f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:40.778026 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 827/1656] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:10:40 2022] DELETE /identity/v3/projects/742e204cb5d249459352af9a37abfeb4 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:40.785524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9eb1ea-996d-4350-9a8e-a8b914bbeb38 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.787530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9eb1ea-996d-4350-9a8e-a8b914bbeb38 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.788728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9eb1ea-996d-4350-9a8e-a8b914bbeb38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.831835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2e9eb1ea-996d-4350-9a8e-a8b914bbeb38 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:40.833245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2e9eb1ea-996d-4350-9a8e-a8b914bbeb38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:40.853177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b042fd7-d72b-45fc-879b-df546153bdec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:40.857376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b042fd7-d72b-45fc-879b-df546153bdec admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.858558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b042fd7-d72b-45fc-879b-df546153bdec admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.859064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b042fd7-d72b-45fc-879b-df546153bdec admin admin] PATH_INFO: `/v3/projects/a5f802ea70504d04af1292c57bb20c58/users/124cd3493db142ce9b127f816ae4bef8/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:40.892503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b042fd7-d72b-45fc-879b-df546153bdec admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5f802ea70504d04af1292c57bb20c58, user_id=124cd3493db142ce9b127f816ae4bef8, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:40.895671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b042fd7-d72b-45fc-879b-df546153bdec admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:40.964255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 830/1657] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:40 2022] PUT /identity/v3/projects/a5f802ea70504d04af1292c57bb20c58/users/124cd3493db142ce9b127f816ae4bef8/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:10:40.976509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:40.976803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:40.977100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.002514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:41.004884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 828/1658] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:41.013843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf8c333-dee1-4806-90b1-a6675f4395be None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.013843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf8c333-dee1-4806-90b1-a6675f4395be None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.013843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf8c333-dee1-4806-90b1-a6675f4395be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.041487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] MFA Rules not processed for user `124cd3493db142ce9b127f816ae4bef8`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:41.042752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:41.048132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8cf8c333-dee1-4806-90b1-a6675f4395be None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:41.049346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8cf8c333-dee1-4806-90b1-a6675f4395be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:41.164023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 829/1659] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:10:41.168052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf8c333-dee1-4806-90b1-a6675f4395be None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.185284 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 831/1660] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:40 2022] POST /identity/v3/auth/tokens => generated 3226 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:41.187122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.188577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:41.193569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf8c333-dee1-4806-90b1-a6675f4395be None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.207412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.208333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.208785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.209245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.210119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:41.210583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.212393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11493b20-e58d-4750-9f50-b93126fa5050 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.214521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.218719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.219994 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.220287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.220568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.221271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.223896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.241835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 830/1661] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:41.244999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.273425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce715b3-409e-4429-b22f-ea0864c19107 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.282261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.284341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.285066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.285340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.290768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.295292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.299561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 832/1662] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:41 2022] POST /identity/v3/projects => generated 365 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:41.303039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.357114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 831/1663] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:41 2022] POST /identity/v3/projects => generated 405 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:41.361382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.444026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e302a-55c0-41e3-90c0-b1584cd8ecd4 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.448643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:INSERT=2,keystone:DELETE=3,keystone:SELECT=95 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:41.482916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7cfe1-2582-478b-8f86-e2dcea9ca27b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.500590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.501654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.502383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.503595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.504906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.507413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.547153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.547996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.548387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.548758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.549618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.552148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.556197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 832/1664] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:41 2022] POST /identity/v3/users => generated 303 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:10:41.559906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.589906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 833/1665] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:41 2022] POST /identity/v3/users => generated 289 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:10:41.590968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.592463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfea8a8-104f-4ce8-84df-a639f4ad0c20 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.598714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.599780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.600446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.601055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.602208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.604421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0abaa-7837-4905-9374-cb73fcc6dd24 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.612176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.616568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.617413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.617612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.617891 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.618615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.620786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.627603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:41.636779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 833/1666] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:41.640779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 834/1667] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:41.642040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.644383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.657123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2686dcd-c482-45d7-a0d4-d4b08123bd7e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.660856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196c6f46-4c65-4990-98d3-c33bdf8d3d94 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.669733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.671042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.671501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.671899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.672779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.673552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.674586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.676911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.677426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.677879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] PATH_INFO: `/v3/projects/8989d038029543bfbba171d5bde605dc/users/fc091deb473841deb7569a8c82828481/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.694291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 835/1668] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:41.698337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.706773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8989d038029543bfbba171d5bde605dc, user_id=fc091deb473841deb7569a8c82828481, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.711770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.725608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7f8876-e51e-460d-b919-9f2b94beff45 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.740484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f2841e-b7ea-408c-82c0-1ce2ef9918ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:41.740484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2841e-b7ea-408c-82c0-1ce2ef9918ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.740484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2841e-b7ea-408c-82c0-1ce2ef9918ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.740484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f2841e-b7ea-408c-82c0-1ce2ef9918ec admin admin] PATH_INFO: `/v3/projects/2437e7bca09446dbb8febc342d7041fd/users/3f1f545993da432e90090e9cb8fba8b6/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.757896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 834/1669] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:41 2022] PUT /identity/v3/projects/8989d038029543bfbba171d5bde605dc/users/fc091deb473841deb7569a8c82828481/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:41.763474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.765835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f2841e-b7ea-408c-82c0-1ce2ef9918ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=2437e7bca09446dbb8febc342d7041fd, user_id=3f1f545993da432e90090e9cb8fba8b6, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:41.771636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f2841e-b7ea-408c-82c0-1ce2ef9918ec admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:41.803787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 836/1670] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:41 2022] PUT /identity/v3/projects/2437e7bca09446dbb8febc342d7041fd/users/3f1f545993da432e90090e9cb8fba8b6/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:10:41.809398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:41.809692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:41.810897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:41.856071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:41.857494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:41.914571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f96323b-3e13-4d14-bdc8-00dcce85530f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:41.967385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 837/1671] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:41.967385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:41.970563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:41.987925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1635d77-fa20-4d4f-80a5-30530ed42e4b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:42.020413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:42.021353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.024429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.024790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] PATH_INFO: `/v3/projects/8757ee77a1884f24a4bddbd2f86ff27d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.026652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8757ee77a1884f24a4bddbd2f86ff27d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:42.029120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:42.055871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:42.056654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.059858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.060268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.062252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:42.063780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:42.098369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 835/1672] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:42.099457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:42.128537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bce58f-72ef-4465-b6ce-33a8f0044493 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:42.151495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8757ee77a1884f24a4bddbd2f86ff27d', 'request_id': 'req-afa98306-5a45-4fe5-a2b0-ec35440c1417'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:42.184388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8757ee77a1884f24a4bddbd2f86ff27d', 'request_id': 'req-afa98306-5a45-4fe5-a2b0-ec35440c1417'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:42.202716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8757ee77a1884f24a4bddbd2f86ff27d', 'request_id': 'req-afa98306-5a45-4fe5-a2b0-ec35440c1417'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:42.235580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] The token cache is being invalidate because project 8757ee77a1884f24a4bddbd2f86ff27d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:10:42.236060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:42.238776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:42.246132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 838/1673] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:10:41 2022] DELETE /identity/v3/projects/8757ee77a1884f24a4bddbd2f86ff27d => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:42.252118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:42.255222 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:42.273737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:42.342009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fa165b-9a60-4b39-970f-500ad441dc8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:42.342803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa165b-9a60-4b39-970f-500ad441dc8a admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.343131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa165b-9a60-4b39-970f-500ad441dc8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.343425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fa165b-9a60-4b39-970f-500ad441dc8a admin admin] PATH_INFO: `/v3/projects/8989d038029543bfbba171d5bde605dc/users/fc091deb473841deb7569a8c82828481/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.361627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fa165b-9a60-4b39-970f-500ad441dc8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=8989d038029543bfbba171d5bde605dc, user_id=fc091deb473841deb7569a8c82828481, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:42.364415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fa165b-9a60-4b39-970f-500ad441dc8a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:42.417117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 836/1674] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:42 2022] PUT /identity/v3/projects/8989d038029543bfbba171d5bde605dc/users/fc091deb473841deb7569a8c82828481/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:42.418444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c786b2b-9eaf-4403-a7e8-4add6ae79f44 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.418738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c786b2b-9eaf-4403-a7e8-4add6ae79f44 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.421086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c786b2b-9eaf-4403-a7e8-4add6ae79f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.458650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:42.477850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa98306-5a45-4fe5-a2b0-ec35440c1417 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:42.491856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8c786b2b-9eaf-4403-a7e8-4add6ae79f44 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:42.493289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8c786b2b-9eaf-4403-a7e8-4add6ae79f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:42.608127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d60f4d-fda1-4bec-b0d0-b5689490f0d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:42.609438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d60f4d-fda1-4bec-b0d0-b5689490f0d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.610021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d60f4d-fda1-4bec-b0d0-b5689490f0d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.610485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d60f4d-fda1-4bec-b0d0-b5689490f0d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.613148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-39d60f4d-fda1-4bec-b0d0-b5689490f0d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:42.661932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 837/1675] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:42.671528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.671822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.672104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.713485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] MFA Rules not processed for user `fc091deb473841deb7569a8c82828481`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:42.714724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:42.775894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d60f4d-fda1-4bec-b0d0-b5689490f0d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:42.779727 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d60f4d-fda1-4bec-b0d0-b5689490f0d2 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:42.797634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 839/1676] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 551 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:42.799452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bdff1-698e-446d-9951-addd88aa42d4 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.799747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bdff1-698e-446d-9951-addd88aa42d4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.800122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bdff1-698e-446d-9951-addd88aa42d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.823505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 838/1677] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:42 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:42.828587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-051bdff1-698e-446d-9951-addd88aa42d4 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:42.830623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-051bdff1-698e-446d-9951-addd88aa42d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:42.849680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 840/1678] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:42.849984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:42.849984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:42.853201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bdff1-698e-446d-9951-addd88aa42d4 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:42.871385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bdff1-698e-446d-9951-addd88aa42d4 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:42.888362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:42.889067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.889354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.889638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.891043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:42.891372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:42.893031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:42.933954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 841/1679] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:10:42.935785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:42.958297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f7080-5262-4364-81cf-2cae5a3ecabb admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:42.971881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:42.972541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:42.972877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:42.973103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:42.973950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:42.980088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:43.012537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df7ca90-5d86-4b47-befc-c4df7de10a75 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:43.031163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4035364-7b64-4142-8a23-8c5b744f0220 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:43.032882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4035364-7b64-4142-8a23-8c5b744f0220 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.033382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4035364-7b64-4142-8a23-8c5b744f0220 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.034091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4035364-7b64-4142-8a23-8c5b744f0220 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.044879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 842/1680] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:42 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:43.048052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:43.065230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4035364-7b64-4142-8a23-8c5b744f0220 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:43.072374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4035364-7b64-4142-8a23-8c5b744f0220 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:43.099172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 839/1681] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:42 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:43.241197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df2fcd-63ef-407e-a5da-4ad697041526 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:43.259420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:43.260099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.260385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.260676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.261432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:43.265825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:43.284886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee68b12-aca8-47e0-84f5-d43178bea603 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.285172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee68b12-aca8-47e0-84f5-d43178bea603 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.285449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee68b12-aca8-47e0-84f5-d43178bea603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.294621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:43.340540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 843/1682] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:43 2022] POST /identity/v3/users => generated 289 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:10:43.342191 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:43.361071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:43.383916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7ee68b12-aca8-47e0-84f5-d43178bea603 None None] MFA Rules not processed for user `1a1125811ba549e0ae66b26ad3949f23`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:43.387534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8b9e04-3feb-47ee-98ae-f53d23782a97 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:43.388723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7ee68b12-aca8-47e0-84f5-d43178bea603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:43.421429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:43.423401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.424174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.426877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.431086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:43.435953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:43.489037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 844/1683] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:43 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:43.493654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:43.525529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5968f06-ad3d-4e7c-9ada-16dbd4852691 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:43.540992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:43.557947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.559351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.561397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.562756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:43.568580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:43.582122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 840/1684] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:43 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:43.584033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.588014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.590082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.627951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 845/1685] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:43 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:43.628914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:43.648537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe00bb1-52c0-4b8a-9d30-a7051f32960f admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:43.657319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53c85fd-a063-43ff-851a-1f9f06294a67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:43.658035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53c85fd-a063-43ff-851a-1f9f06294a67 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.658534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] MFA Rules not processed for user `1a1125811ba549e0ae66b26ad3949f23`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:43.659337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53c85fd-a063-43ff-851a-1f9f06294a67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.659629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53c85fd-a063-43ff-851a-1f9f06294a67 admin admin] PATH_INFO: `/v3/projects/3582316ee9b9472a9fda807af4b25c62/users/acc5e42ec3e9424db26987a312f66a44/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.660818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:43.683816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 841/1686] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:43 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:43.690522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:43.692146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:43.705899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53c85fd-a063-43ff-851a-1f9f06294a67 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3582316ee9b9472a9fda807af4b25c62, user_id=acc5e42ec3e9424db26987a312f66a44, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:43.716045 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53c85fd-a063-43ff-851a-1f9f06294a67 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:43.755776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 846/1687] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:43 2022] PUT /identity/v3/projects/3582316ee9b9472a9fda807af4b25c62/users/acc5e42ec3e9424db26987a312f66a44/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:43.772623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f4fa9-5b8d-4541-b0d3-87206d6144d6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:43.772931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f4fa9-5b8d-4541-b0d3-87206d6144d6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:43.773189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f4fa9-5b8d-4541-b0d3-87206d6144d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:43.865024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-184f4fa9-5b8d-4541-b0d3-87206d6144d6 None None] MFA Rules not processed for user `acc5e42ec3e9424db26987a312f66a44`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:43.866319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-184f4fa9-5b8d-4541-b0d3-87206d6144d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:43.883749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4274ef-7f69-4d85-9bae-3d612d8d0808 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:44.119297 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:44.120218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:44.120499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:44.120775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:44.137101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 847/1688] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:43 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 365 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:10:44.138074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184f4fa9-5b8d-4541-b0d3-87206d6144d6 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:44.175211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:44.260117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:44.262611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:44.283718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 842/1689] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:43 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 594 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:10:44.360317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184f4fa9-5b8d-4541-b0d3-87206d6144d6 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:44.365301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:44.365967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:44.366232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:44.366597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:44.367247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:44.369178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:44.438839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:44.439301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 848/1690] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:44 2022] POST /identity/v3/projects => generated 377 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:44.440299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:44.451486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:44.457282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ef3da0-bff2-4904-95fb-e30027d69738 service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:44.529640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:44.638332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6e611b-7f67-4a03-a6e5-d2806cc071e2 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:44.650907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:44.651543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:44.651825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:44.652103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] PATH_INFO: `/v3/users/124cd3493db142ce9b127f816ae4bef8` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:44.652526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:44.653862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:44.654843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:44.655322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:44.655753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] RBAC: Authorizing `identity:delete_user(user_id=124cd3493db142ce9b127f816ae4bef8)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:44.656511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:44.657696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:44.660748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:44.703823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 843/1691] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:44 2022] POST /identity/v3/users => generated 295 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:10:44.708260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:44.735952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c496d0d-a9f1-4a6c-8fc5-03d7474874cf admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:44.747245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:44.748933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:44.751195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:44.751636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:44.753988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:44.756096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:44.762180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '124cd3493db142ce9b127f816ae4bef8'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:44.780527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 844/1692] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:44 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:44.786383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:44.786837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '124cd3493db142ce9b127f816ae4bef8'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:44.795879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '124cd3493db142ce9b127f816ae4bef8'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:44.811362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '124cd3493db142ce9b127f816ae4bef8'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:44.812573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b89799c-a031-411a-8930-e53975a57e40 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '124cd3493db142ce9b127f816ae4bef8'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:44.816436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f4b70d-093e-4d07-984e-f571099f7c69 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:44.825743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 849/1693] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:10:44 2022] DELETE /identity/v3/users/124cd3493db142ce9b127f816ae4bef8 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:44.828094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:44.828383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:44.828664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:44.836880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:44.837637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:44.837918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:44.838192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] PATH_INFO: `/v3/projects/4e81ec3b6314454c9201ab5d935d24fd/users/e7193ed031a54d6bbe2c1c174d9ef997/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:44.872054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:44.874388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e81ec3b6314454c9201ab5d935d24fd, user_id=e7193ed031a54d6bbe2c1c174d9ef997, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:44.877774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:44.880113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:44.907307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 845/1694] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:44 2022] PUT /identity/v3/projects/4e81ec3b6314454c9201ab5d935d24fd/users/e7193ed031a54d6bbe2c1c174d9ef997/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:44.911386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:44.914113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:44.963079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 850/1695] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:45.013323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:45.015197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:45.057326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefa1ef6-99cc-4cff-b5a4-aaed8f7d93fb admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:45.068922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:45.074004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.074289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.074568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.075328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:45.087362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:45.115810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 846/1696] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:44 2022] GET /identity/v3/roles => generated 1420 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:45.120024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:45.147977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830561bb-3323-494a-b47d-d863fee4be36 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:45.152635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebb64ee-f16e-4e48-8136-ba541df90c84 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:45.159072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a327d1-6e59-4ea5-83d3-7a74ea0e9bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:45.159785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a327d1-6e59-4ea5-83d3-7a74ea0e9bc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.160070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a327d1-6e59-4ea5-83d3-7a74ea0e9bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.160462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a327d1-6e59-4ea5-83d3-7a74ea0e9bc8 admin admin] PATH_INFO: `/v3/projects/4e81ec3b6314454c9201ab5d935d24fd/users/e7193ed031a54d6bbe2c1c174d9ef997/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.164402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:45.168342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a327d1-6e59-4ea5-83d3-7a74ea0e9bc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e81ec3b6314454c9201ab5d935d24fd, user_id=e7193ed031a54d6bbe2c1c174d9ef997, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:45.171733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a327d1-6e59-4ea5-83d3-7a74ea0e9bc8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:45.186866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:45.198436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 847/1697] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:45 2022] PUT /identity/v3/projects/4e81ec3b6314454c9201ab5d935d24fd/users/e7193ed031a54d6bbe2c1c174d9ef997/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:45.209235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.209525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.209801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.265055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad7d1a1-cbfa-4110-bca0-fd217e62fe31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:45.265701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad7d1a1-cbfa-4110-bca0-fd217e62fe31 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.265984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad7d1a1-cbfa-4110-bca0-fd217e62fe31 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.266260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad7d1a1-cbfa-4110-bca0-fd217e62fe31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.267756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] MFA Rules not processed for user `e7193ed031a54d6bbe2c1c174d9ef997`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:45.269656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:45.297469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad7d1a1-cbfa-4110-bca0-fd217e62fe31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:45.298969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad7d1a1-cbfa-4110-bca0-fd217e62fe31 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:45.324600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 851/1698] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:10:45.329707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e189e591-9903-4f3d-97ba-a5d4ec5a7091 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.330004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e189e591-9903-4f3d-97ba-a5d4ec5a7091 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.330287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e189e591-9903-4f3d-97ba-a5d4ec5a7091 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.331894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 852/1699] 10.222.0.42 () {54 vars in 912 bytes} [Tue Jul 26 03:10:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:10:45.342380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd9c92-2816-417d-bd99-5c192d32e206 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.342657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd9c92-2816-417d-bd99-5c192d32e206 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.342939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd9c92-2816-417d-bd99-5c192d32e206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.463330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-11cd9c92-2816-417d-bd99-5c192d32e206 None None] MFA Rules not processed for user `29a1868eb95f4396aa3fd2da677b40f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:45.464593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-11cd9c92-2816-417d-bd99-5c192d32e206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:45.481288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 848/1700] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:45 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:10:45.493688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:45.495485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:45.634546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 853/1701] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:10:45 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:45.643200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd081f9-75b4-476a-9be5-cca1b3a16259 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:45.647528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bb5ad0-e754-407e-aa08-4e5eee0996e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:45.648126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bb5ad0-e754-407e-aa08-4e5eee0996e0 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.648404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bb5ad0-e754-407e-aa08-4e5eee0996e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.649248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bb5ad0-e754-407e-aa08-4e5eee0996e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.650774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-96bb5ad0-e754-407e-aa08-4e5eee0996e0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:45.732030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bb5ad0-e754-407e-aa08-4e5eee0996e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:45.733608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bb5ad0-e754-407e-aa08-4e5eee0996e0 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:45.745290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 849/1702] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:45 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:45.899031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd9c92-2816-417d-bd99-5c192d32e206 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:45.927934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd9c92-2816-417d-bd99-5c192d32e206 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:45.946368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:45.949700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:45.950204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:45.951965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:45.982645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:45.987029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:46.005129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 854/1703] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:45 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:46.715949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173f2f34-3377-4768-8aef-326c75c61c8f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:46.715949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173f2f34-3377-4768-8aef-326c75c61c8f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:46.715949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173f2f34-3377-4768-8aef-326c75c61c8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:46.812451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-173f2f34-3377-4768-8aef-326c75c61c8f None None] MFA Rules not processed for user `037c9f8229a44de198ebc3752f7a1966`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:46.819479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-173f2f34-3377-4768-8aef-326c75c61c8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:46.926721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:47.009102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 850/1704] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:46 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:47.019242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:47.024525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:47.140522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a4051-7e0a-4155-a346-6c9b35df2dfd service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:47.156625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad93218a-d6b5-4e9e-ba49-8b65e345b19b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:47.162177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad93218a-d6b5-4e9e-ba49-8b65e345b19b service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:47.162463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad93218a-d6b5-4e9e-ba49-8b65e345b19b service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:47.162899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad93218a-d6b5-4e9e-ba49-8b65e345b19b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:47.176115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad93218a-d6b5-4e9e-ba49-8b65e345b19b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:47.179082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad93218a-d6b5-4e9e-ba49-8b65e345b19b service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:47.200527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 855/1705] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:47.335432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec9b3aa-8601-4499-815f-fa75070786e6 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:47.335884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec9b3aa-8601-4499-815f-fa75070786e6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:47.336263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec9b3aa-8601-4499-815f-fa75070786e6 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:47.337770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 851/1706] 10.222.0.42 () {54 vars in 961 bytes} [Tue Jul 26 03:10:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:10:47.346159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f33840-d6af-4e7d-a17a-16e069686673 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:47.346585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f33840-d6af-4e7d-a17a-16e069686673 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:47.347288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f33840-d6af-4e7d-a17a-16e069686673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:47.397087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f6f33840-d6af-4e7d-a17a-16e069686673 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:47.398214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f6f33840-d6af-4e7d-a17a-16e069686673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:47.451825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:47.473708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:47.474149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:47.474611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:47.505235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] MFA Rules not processed for user `1a1125811ba549e0ae66b26ad3949f23`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:47.506592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:47.585663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 856/1707] 10.222.0.42 () {58 vars in 1058 bytes} [Tue Jul 26 03:10:47 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:10:47.594031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f33840-d6af-4e7d-a17a-16e069686673 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:47.604190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 852/1708] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:47 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:47.608285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f33840-d6af-4e7d-a17a-16e069686673 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:47.617544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88b02bd-9e27-412f-bc48-7afc1ef25c70 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:47.618035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b02bd-9e27-412f-bc48-7afc1ef25c70 service placement] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:47.618320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b02bd-9e27-412f-bc48-7afc1ef25c70 service placement] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:47.618600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b02bd-9e27-412f-bc48-7afc1ef25c70 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:47.621305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b88b02bd-9e27-412f-bc48-7afc1ef25c70 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:47.642368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88b02bd-9e27-412f-bc48-7afc1ef25c70 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:47.645583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88b02bd-9e27-412f-bc48-7afc1ef25c70 service placement] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:47.664972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 857/1709] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:47 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:10:48.522579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:48.524611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:48.549687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a2d954-e6dc-44c8-9d61-d5feea72029e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:48.550526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a2d954-e6dc-44c8-9d61-d5feea72029e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:48.550977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a2d954-e6dc-44c8-9d61-d5feea72029e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:48.553322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 858/1710] 10.222.0.42 () {54 vars in 912 bytes} [Tue Jul 26 03:10:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:10:48.560053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:48.560544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:48.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:48.603522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] MFA Rules not processed for user `29a1868eb95f4396aa3fd2da677b40f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:48.607530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:48.640999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 859/1711] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:10:48 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:10:48.696877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284adea8-4e68-4cdc-8592-7fc55a322157 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:48.710082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:48.710968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:48.711657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:48.711766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:48.711875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] PATH_INFO: `/v3/projects/a5f802ea70504d04af1292c57bb20c58` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:48.713569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:48.713765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5f802ea70504d04af1292c57bb20c58)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:48.715717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:48.741694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571e2316-8354-4a89-8b1d-1f0a278f565f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:48.754384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:48.756394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:48.758746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:48.760631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:48.773181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:48.775717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:48.788156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 860/1712] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:48 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:48.811674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5f802ea70504d04af1292c57bb20c58', 'request_id': 'req-00e5ac45-b4b6-4564-94ff-7d9596a769d9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:48.827669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5f802ea70504d04af1292c57bb20c58', 'request_id': 'req-00e5ac45-b4b6-4564-94ff-7d9596a769d9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:48.846130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5f802ea70504d04af1292c57bb20c58', 'request_id': 'req-00e5ac45-b4b6-4564-94ff-7d9596a769d9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:48.867362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] The token cache is being invalidate because project a5f802ea70504d04af1292c57bb20c58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:10:48.868153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:48.869305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-00e5ac45-b4b6-4564-94ff-7d9596a769d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:48.873319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 853/1713] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:10:48 2022] DELETE /identity/v3/projects/a5f802ea70504d04af1292c57bb20c58 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:48.877699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:48.896197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f0c08f-134c-450c-b3d8-d1f3752d7a66 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:49.054542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:49.055648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:49.056049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:49.056459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] PATH_INFO: `/v3/users/fc091deb473841deb7569a8c82828481` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:49.057837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc091deb473841deb7569a8c82828481)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:49.060995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:49.177826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc091deb473841deb7569a8c82828481'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:49.203041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc091deb473841deb7569a8c82828481'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:49.214906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc091deb473841deb7569a8c82828481'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:49.231983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc091deb473841deb7569a8c82828481'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:49.233174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc091deb473841deb7569a8c82828481'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:49.236847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:49.243192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 861/1714] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:10:48 2022] DELETE /identity/v3/users/fc091deb473841deb7569a8c82828481 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:49.349928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f590dc-85c9-4b27-9772-6f1d006538cf None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:49.351166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f590dc-85c9-4b27-9772-6f1d006538cf None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:49.351166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f590dc-85c9-4b27-9772-6f1d006538cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:49.361811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:49.412278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-91f590dc-85c9-4b27-9772-6f1d006538cf None None] MFA Rules not processed for user `3f1f545993da432e90090e9cb8fba8b6`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:49.414457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-91f590dc-85c9-4b27-9772-6f1d006538cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:49.586998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 854/1715] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:49 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:49.591654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:49.593487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:49.765762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1ba33a-ef01-4545-b8ea-e00034d0c70f admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:49.776950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:49.777619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:49.777903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:49.778182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:49.782095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:49.791825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:49.885214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 862/1716] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:49 2022] POST /identity/v3/projects => generated 405 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:49.886378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f590dc-85c9-4b27-9772-6f1d006538cf None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:49.905291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f590dc-85c9-4b27-9772-6f1d006538cf None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:50.058793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:50.059823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:50.060325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:50.060764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:50.074120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:50.078552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:50.150258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 855/1717] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:49 2022] POST /identity/v3/users => generated 309 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:10:50.153384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:50.170829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11b43f0-0921-4f82-ae8f-e5cb1f7dd691 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:50.183226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:50.184292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:50.184673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:50.185029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:50.188568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:50.191719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:50.211138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 863/1718] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:50 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:50.215065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:50.230981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351ff178-66d7-4615-b745-e778d19d8a4a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:50.236381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:50.237419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:50.237602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:50.237975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] PATH_INFO: `/v3/projects/afd263fdeaa345bba964bb0b516787e3/users/8e94663a93cf40f896d6e84477595930/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:50.258670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=afd263fdeaa345bba964bb0b516787e3, user_id=8e94663a93cf40f896d6e84477595930, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:50.263065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:50.284124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 856/1719] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:50 2022] PUT /identity/v3/projects/afd263fdeaa345bba964bb0b516787e3/users/8e94663a93cf40f896d6e84477595930/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:50.288915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:50.300734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a80b33-4ce3-483a-a06f-de1a35cad0c6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:50.432435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:50.433094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:50.433372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:50.433826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:50.434634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:50.436630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:50.451432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 864/1720] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:50 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:50.456160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:50.472309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb38dc8f-4101-4ce6-a625-a91e5d32ac1c admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:50.478250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02108474-1bfc-433c-8e0b-6e6206c9e410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:50.479981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02108474-1bfc-433c-8e0b-6e6206c9e410 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:50.481207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02108474-1bfc-433c-8e0b-6e6206c9e410 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:50.481697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02108474-1bfc-433c-8e0b-6e6206c9e410 admin admin] PATH_INFO: `/v3/projects/afd263fdeaa345bba964bb0b516787e3/users/8e94663a93cf40f896d6e84477595930/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:50.484226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02108474-1bfc-433c-8e0b-6e6206c9e410 admin admin] RBAC: Authorizing `identity:create_grant(project_id=afd263fdeaa345bba964bb0b516787e3, user_id=8e94663a93cf40f896d6e84477595930, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:50.488110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02108474-1bfc-433c-8e0b-6e6206c9e410 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:50.504652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 857/1721] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:50 2022] PUT /identity/v3/projects/afd263fdeaa345bba964bb0b516787e3/users/8e94663a93cf40f896d6e84477595930/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:50.980021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:50.982362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:51.031002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ba830e-736e-4cd7-a738-99153f1e78f0 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:51.170741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:51.171476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:51.171770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:51.172048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] PATH_INFO: `/v3/projects/8989d038029543bfbba171d5bde605dc` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:51.174546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=8989d038029543bfbba171d5bde605dc)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:51.176545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:51.230088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:51.335212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8989d038029543bfbba171d5bde605dc', 'request_id': 'req-5c218fcf-8092-4289-ae7c-eed425e3ea3d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:51.363577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8989d038029543bfbba171d5bde605dc', 'request_id': 'req-5c218fcf-8092-4289-ae7c-eed425e3ea3d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:51.371914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8989d038029543bfbba171d5bde605dc', 'request_id': 'req-5c218fcf-8092-4289-ae7c-eed425e3ea3d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:51.428103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] The token cache is being invalidate because project 8989d038029543bfbba171d5bde605dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:10:51.434690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:51.444772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5c218fcf-8092-4289-ae7c-eed425e3ea3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:51.451247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 865/1722] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:10:50 2022] DELETE /identity/v3/projects/8989d038029543bfbba171d5bde605dc => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:51.457007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e47e374-8620-45eb-b5f3-be645b4f351f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:51.457569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e47e374-8620-45eb-b5f3-be645b4f351f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:51.457569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e47e374-8620-45eb-b5f3-be645b4f351f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:51.525355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7e47e374-8620-45eb-b5f3-be645b4f351f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:51.527056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7e47e374-8620-45eb-b5f3-be645b4f351f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:51.650539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:51.712068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 858/1723] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:51.720404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24117622-5283-4bfa-95db-b3d4b0475bb5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:51.720957 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24117622-5283-4bfa-95db-b3d4b0475bb5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:51.721807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24117622-5283-4bfa-95db-b3d4b0475bb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:51.788586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-24117622-5283-4bfa-95db-b3d4b0475bb5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:51.795294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-24117622-5283-4bfa-95db-b3d4b0475bb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:51.825314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 866/1724] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:51.825463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e47e374-8620-45eb-b5f3-be645b4f351f None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:51.844590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e47e374-8620-45eb-b5f3-be645b4f351f None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:51.863168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:51.863962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:51.864325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:51.864827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:51.866908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:51.869009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:51.870858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:51.903203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 859/1725] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:10:51.905739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24117622-5283-4bfa-95db-b3d4b0475bb5 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:51.918649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24117622-5283-4bfa-95db-b3d4b0475bb5 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:51.937322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:51.939425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:51.941473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:51.942539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:51.947857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:51.956144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:52.062309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 867/1726] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:51 2022] POST /identity/v3/projects => generated 379 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:52.067092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:52.079419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275869e6-a952-4e47-b33e-d2c8c8659bd8 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:52.246488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:52.247372 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:52.248319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:52.249869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:52.252093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:52.258518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:52.320710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 860/1727] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:52 2022] POST /identity/v3/users => generated 290 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:52.324623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:52.341528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3726f86d-1d73-43d7-b4d2-2352facc6388 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:52.354332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:52.354870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:52.355178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:52.355455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:52.356392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:52.360332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:52.406884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:52.425494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 868/1728] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:52 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:52.431977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0311ccf-14a1-4a53-afae-c50472b542ca admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:52.448977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6441fed5-ac66-47f1-b2b0-b86a2d7b94a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:52.449627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6441fed5-ac66-47f1-b2b0-b86a2d7b94a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:52.449910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6441fed5-ac66-47f1-b2b0-b86a2d7b94a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:52.450234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6441fed5-ac66-47f1-b2b0-b86a2d7b94a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:52.451234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6441fed5-ac66-47f1-b2b0-b86a2d7b94a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:52.454737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6441fed5-ac66-47f1-b2b0-b86a2d7b94a7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:52.505487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:52.511404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 861/1729] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:52 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:52.538861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17df223-fd80-449d-9e35-e2c36db48684 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:52.541079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:52.541363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:52.541641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:52.552112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d39406-75f2-4522-b381-97d1037c7613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:52.552695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d39406-75f2-4522-b381-97d1037c7613 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:52.552989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d39406-75f2-4522-b381-97d1037c7613 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:52.555511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d39406-75f2-4522-b381-97d1037c7613 admin admin] PATH_INFO: `/v3/projects/fd742c3aab8945ffb77d07429bbd2dd4/users/25b7e79a824d47deac0f11e3cf86982e/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:52.608626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] MFA Rules not processed for user `037c9f8229a44de198ebc3752f7a1966`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:52.609950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:52.621177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d39406-75f2-4522-b381-97d1037c7613 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd742c3aab8945ffb77d07429bbd2dd4, user_id=25b7e79a824d47deac0f11e3cf86982e, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:52.634990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d39406-75f2-4522-b381-97d1037c7613 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:52.692620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 869/1730] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:52 2022] PUT /identity/v3/projects/fd742c3aab8945ffb77d07429bbd2dd4/users/25b7e79a824d47deac0f11e3cf86982e/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:52.696355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa8a51-3219-4bd0-a4fa-e683ac16e311 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:52.696661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa8a51-3219-4bd0-a4fa-e683ac16e311 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:52.696946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fa8a51-3219-4bd0-a4fa-e683ac16e311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:52.748068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 862/1731] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:10:52.752888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:52.755599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:52.755821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-75fa8a51-3219-4bd0-a4fa-e683ac16e311 None None] MFA Rules not processed for user `25b7e79a824d47deac0f11e3cf86982e`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:52.757209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-75fa8a51-3219-4bd0-a4fa-e683ac16e311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:52.930303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b89f845-4989-4316-aa77-d702f9150f23 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:52.937594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 870/1732] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:52 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:52.942772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa8a51-3219-4bd0-a4fa-e683ac16e311 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:52.964745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:53.039356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:53.124090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:53.124731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:53.135286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:53.136556 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:53.144873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:53.158775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fa8a51-3219-4bd0-a4fa-e683ac16e311 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:53.163432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:53.164290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:53.164709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:53.165126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:53.165830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:53.167834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:53.223697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 871/1733] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:52 2022] POST /identity/v3/projects => generated 381 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:10:53.228049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:53.320938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:53.322480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:53.368333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 863/1734] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:52 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 619 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:53.425162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e1860-888c-4176-8ddd-65e5b1d37ef0 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:53.445870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:53.446716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:53.447052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:53.447312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:53.448019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:53.449997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:53.523813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 872/1735] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:10:53 2022] POST /identity/v3/users => generated 297 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:53.528178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:53.548961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a225b0-a7e1-451f-9cba-397cc63016ff service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:53.564612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:53.568425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:53.568621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:53.568806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:53.570488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:53.576836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:53.622935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 864/1736] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:53 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:53.624885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:53.647422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04441cb2-9ff0-4bcd-b6f6-940e2240faf2 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:53.661292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c6c332-7d0a-439a-a68f-37e2c5a21b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:53.662008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6c332-7d0a-439a-a68f-37e2c5a21b5d admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:53.662291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6c332-7d0a-439a-a68f-37e2c5a21b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:53.662569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c6c332-7d0a-439a-a68f-37e2c5a21b5d admin admin] PATH_INFO: `/v3/projects/e6d19b54094d4429b07087b208d9b78f/users/502f968e8bae49d5afb8eb10ca32626c/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:53.709171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c6c332-7d0a-439a-a68f-37e2c5a21b5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6d19b54094d4429b07087b208d9b78f, user_id=502f968e8bae49d5afb8eb10ca32626c, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:53.711930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c6c332-7d0a-439a-a68f-37e2c5a21b5d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:53.766852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 873/1737] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:53 2022] PUT /identity/v3/projects/e6d19b54094d4429b07087b208d9b78f/users/502f968e8bae49d5afb8eb10ca32626c/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:53.767610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:53.782168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba11f6a8-66ef-4694-be23-7bea79f81fb0 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:53.806712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedd1d39-3611-48c4-baec-a931914be63c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:53.807056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedd1d39-3611-48c4-baec-a931914be63c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:53.807333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedd1d39-3611-48c4-baec-a931914be63c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:53.991589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-aedd1d39-3611-48c4-baec-a931914be63c None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:53.992843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-aedd1d39-3611-48c4-baec-a931914be63c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:54.029108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:54.032405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:54.037164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:54.037653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:54.043877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:54.045856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:54.088284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 865/1738] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:10:53 2022] GET /identity/v3/roles => generated 1420 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:10:54.089435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:54.135795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042ab78c-aa2d-497a-adfa-5cb839b2e7a7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:54.164668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e211fb-7973-4b87-8599-01ed9ed44446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:54.165607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e211fb-7973-4b87-8599-01ed9ed44446 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:54.166021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e211fb-7973-4b87-8599-01ed9ed44446 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:54.166411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e211fb-7973-4b87-8599-01ed9ed44446 admin admin] PATH_INFO: `/v3/projects/e6d19b54094d4429b07087b208d9b78f/users/502f968e8bae49d5afb8eb10ca32626c/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:54.175857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e211fb-7973-4b87-8599-01ed9ed44446 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6d19b54094d4429b07087b208d9b78f, user_id=502f968e8bae49d5afb8eb10ca32626c, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:54.178769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e211fb-7973-4b87-8599-01ed9ed44446 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:54.190858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:54.217425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 866/1739] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:10:54 2022] PUT /identity/v3/projects/e6d19b54094d4429b07087b208d9b78f/users/502f968e8bae49d5afb8eb10ca32626c/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:54.230588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:54.246933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:54.247843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:54.278785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 874/1740] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:10:53 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 473 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:54.368581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] MFA Rules not processed for user `502f968e8bae49d5afb8eb10ca32626c`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:54.374416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:54.423111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:54.528233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 867/1741] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:54 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:54.542508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedd1d39-3611-48c4-baec-a931914be63c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:54.566303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:54.567728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:54.642160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedd1d39-3611-48c4-baec-a931914be63c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:54.723768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:54.724434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:54.724716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:54.724991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:54.745059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:54.746513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:54.788607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7098df-2a61-4c0b-9eee-5d775364d3cb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:54.797706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 875/1742] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:54 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:54.814180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:54.814843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:54.815155 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:54.815435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:54.953174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:54.955555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:54.994655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 868/1743] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:10:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 26 03:10:55.583912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:55.604771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fc079-ccc1-4f8b-8a9e-c6db0c4690f0 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:55.633021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '502f968e8bae49d5afb8eb10ca32626c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e6d19b54094d4429b07087b208d9b78f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:55.634495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:55.634792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:55.635101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:55.642601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:55.643060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:55.648474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:55.737618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 876/1744] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:55.737726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:55.756281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947f8dd7-f383-4c78-a979-9d835fc4d760 service swift] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:55.780293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669fbde1-eeda-4a93-be9e-30ccb1560843 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '502f968e8bae49d5afb8eb10ca32626c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e6d19b54094d4429b07087b208d9b78f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:55.781214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbde1-eeda-4a93-be9e-30ccb1560843 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:55.783327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbde1-eeda-4a93-be9e-30ccb1560843 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:55.784916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fbde1-eeda-4a93-be9e-30ccb1560843 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:55.787209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669fbde1-eeda-4a93-be9e-30ccb1560843 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:55.791679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669fbde1-eeda-4a93-be9e-30ccb1560843 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:55.951650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:55.953610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 869/1745] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:10:55 2022] POST /identity/v3/projects => generated 354 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:10:55.977805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea075640-5e7b-4f93-8d04-7155c018a7cc tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:56.147894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ed7a5-3c16-4e8b-a214-46e71ca206a1 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:56.148432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ed7a5-3c16-4e8b-a214-46e71ca206a1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:56.148909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70ed7a5-3c16-4e8b-a214-46e71ca206a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:56.203112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '502f968e8bae49d5afb8eb10ca32626c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e6d19b54094d4429b07087b208d9b78f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:56.204183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:56.204466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:56.206809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:56.208841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:10:56.208841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:56.210190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:56.227263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 877/1746] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:10:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:10:56.246239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b70ed7a5-3c16-4e8b-a214-46e71ca206a1 None None] MFA Rules not processed for user `fbdac72f58df4335b3790c5fb1609f5b`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:56.247687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b70ed7a5-3c16-4e8b-a214-46e71ca206a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:56.286090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:56.296828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:56.325526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38df027-0327-44e5-92c9-b4c9db45f695 tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:56.333093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '502f968e8bae49d5afb8eb10ca32626c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e6d19b54094d4429b07087b208d9b78f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:56.333771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:56.334057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:56.334516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] PATH_INFO: `/v3/projects/b56c8ad72b0a4d2d871fd974851e40a1` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:56.335685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorizing `identity:delete_project(project_id=b56c8ad72b0a4d2d871fd974851e40a1)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:56.338081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:56.373272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:56.422250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b56c8ad72b0a4d2d871fd974851e40a1', 'request_id': 'req-bc89e332-11a8-4ca1-8d04-227f790cce4f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:56.446061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b56c8ad72b0a4d2d871fd974851e40a1', 'request_id': 'req-bc89e332-11a8-4ca1-8d04-227f790cce4f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:56.455400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 870/1747] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:56 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:56.458744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:56.459064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:56.459374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:56.460738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b56c8ad72b0a4d2d871fd974851e40a1', 'request_id': 'req-bc89e332-11a8-4ca1-8d04-227f790cce4f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:56.499520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] The token cache is being invalidate because project b56c8ad72b0a4d2d871fd974851e40a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:10:56.499850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:56.500679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:56.505828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 878/1748] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:10:56 2022] DELETE /identity/v3/projects/b56c8ad72b0a4d2d871fd974851e40a1 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:56.505828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:56.505828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:56.545555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] MFA Rules not processed for user `8e94663a93cf40f896d6e84477595930`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:56.547061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:56.627347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc89e332-11a8-4ca1-8d04-227f790cce4f tempest-VolumeQuotaClassesTest-2024008183 tempest-VolumeQuotaClassesTest-2024008183-project-admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:56.636841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642c34f1-017d-43a1-b648-ca3fd4a671ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:56.637738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c34f1-017d-43a1-b648-ca3fd4a671ae service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:56.638171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c34f1-017d-43a1-b648-ca3fd4a671ae service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:56.638742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c34f1-017d-43a1-b648-ca3fd4a671ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:56.642063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-642c34f1-017d-43a1-b648-ca3fd4a671ae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:56.699440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 871/1749] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:56 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:56.721903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642c34f1-017d-43a1-b648-ca3fd4a671ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:56.723618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642c34f1-017d-43a1-b648-ca3fd4a671ae service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:56.747147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 879/1750] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:56 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:10:56.951635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:56.953122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:57.130808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e27f99-cdb2-434c-ab52-c95757985b32 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:57.160507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:57.162445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:57.162990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:57.163636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] PATH_INFO: `/v3/users/25b7e79a824d47deac0f11e3cf86982e` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:57.165504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] RBAC: Authorizing `identity:delete_user(user_id=25b7e79a824d47deac0f11e3cf86982e)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:57.168583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:57.311351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:57.351533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25b7e79a824d47deac0f11e3cf86982e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:57.375507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25b7e79a824d47deac0f11e3cf86982e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:57.421717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25b7e79a824d47deac0f11e3cf86982e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:57.445262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25b7e79a824d47deac0f11e3cf86982e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:57.446249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4c334ff9-9ce2-421a-9675-195f163d3e7a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25b7e79a824d47deac0f11e3cf86982e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:10:57.454533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 872/1751] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:10:56 2022] DELETE /identity/v3/users/25b7e79a824d47deac0f11e3cf86982e => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:10:57.460867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9903a7f4-90a5-4c18-8347-d7d4a7d91622 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:57.461297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9903a7f4-90a5-4c18-8347-d7d4a7d91622 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:57.461686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9903a7f4-90a5-4c18-8347-d7d4a7d91622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:57.505039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-9903a7f4-90a5-4c18-8347-d7d4a7d91622 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:57.511343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9903a7f4-90a5-4c18-8347-d7d4a7d91622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:57.644691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:10:57.691997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 880/1752] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:10:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:58.473997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a676f621-813a-4ce8-919a-baf686e5c65c None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:58.476614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a676f621-813a-4ce8-919a-baf686e5c65c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:58.477441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a676f621-813a-4ce8-919a-baf686e5c65c None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:58.481468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 873/1753] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:10:58 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:10:58.518238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a953248-a608-4854-9ece-251585d13e84 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:58.518940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a953248-a608-4854-9ece-251585d13e84 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:58.519607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a953248-a608-4854-9ece-251585d13e84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:58.555936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a676f621-813a-4ce8-919a-baf686e5c65c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:58.555936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a676f621-813a-4ce8-919a-baf686e5c65c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:58.589105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5a953248-a608-4854-9ece-251585d13e84 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:10:58.589105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5a953248-a608-4854-9ece-251585d13e84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:58.867397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a676f621-813a-4ce8-919a-baf686e5c65c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:58.879845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f028bfbf-dde3-4db6-95ec-519b2c19fd12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:58.884036 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028bfbf-dde3-4db6-95ec-519b2c19fd12 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:58.884357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028bfbf-dde3-4db6-95ec-519b2c19fd12 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:58.884614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f028bfbf-dde3-4db6-95ec-519b2c19fd12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:58.980771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 881/1754] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:10:58 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:10:58.981433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f028bfbf-dde3-4db6-95ec-519b2c19fd12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:58.999334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f028bfbf-dde3-4db6-95ec-519b2c19fd12 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:59.049192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a953248-a608-4854-9ece-251585d13e84 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:10:59.052733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5a953248-a608-4854-9ece-251585d13e84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:10:59.087574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 874/1755] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 535 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:10:59.105859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a953248-a608-4854-9ece-251585d13e84 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:10:59.156909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:10:59.158462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:10:59.158751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:10:59.175693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:10:59.232938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:10:59.244973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:10:59.298744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 882/1756] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:10:59 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:00.700543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a70e77-1397-4a1e-8844-f92d612ac363 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:00.701624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a70e77-1397-4a1e-8844-f92d612ac363 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:00.702024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a70e77-1397-4a1e-8844-f92d612ac363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:00.806298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-29a70e77-1397-4a1e-8844-f92d612ac363 None None] MFA Rules not processed for user `29a1868eb95f4396aa3fd2da677b40f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:00.816754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-29a70e77-1397-4a1e-8844-f92d612ac363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:00.858834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 875/1757] 10.222.0.36 () {58 vars in 1015 bytes} [Tue Jul 26 03:11:00 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:00.936612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:00.974000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc942e8e-13ef-4ba8-a35b-16153a353341 service placement] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:00.999324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702a267-2e70-4e85-a808-ae23eaa2361f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:01.001870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702a267-2e70-4e85-a808-ae23eaa2361f service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:01.002795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702a267-2e70-4e85-a808-ae23eaa2361f service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:01.019787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702a267-2e70-4e85-a808-ae23eaa2361f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:01.039703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702a267-2e70-4e85-a808-ae23eaa2361f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:01.042675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702a267-2e70-4e85-a808-ae23eaa2361f service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:01.078992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 883/1758] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:00 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:01.600378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:01.600820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:01.601246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:01.668637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] MFA Rules not processed for user `adcae81584cc447db8d2628ff6e1bf5b`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:01.670119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:01.902794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 876/1759] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:01 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:01.913679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30b60b-2c67-476e-b504-857460736f9c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:01.914331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30b60b-2c67-476e-b504-857460736f9c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:01.914786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30b60b-2c67-476e-b504-857460736f9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:01.952834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-9a30b60b-2c67-476e-b504-857460736f9c None None] MFA Rules not processed for user `adcae81584cc447db8d2628ff6e1bf5b`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:01.956118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9a30b60b-2c67-476e-b504-857460736f9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:01.976945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 884/1760] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:01 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:02.021470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:02.023357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:02.159137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:02.270868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536bc308-2707-4bb1-9565-6ee60c78a0ae None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:02.287489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:02.289907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:02.290691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:02.291632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:02.316501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:02.320078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:02.374421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 877/1761] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:02 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:03.666806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc77929-a6d6-454a-854f-702df0fe657c None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:03.668058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc77929-a6d6-454a-854f-702df0fe657c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:03.670050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc77929-a6d6-454a-854f-702df0fe657c None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:03.672592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 885/1762] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:11:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:11:03.711868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:03.713382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:03.720041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7591140-151e-45bc-8fb0-56cc1e193d41 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:03.720491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7591140-151e-45bc-8fb0-56cc1e193d41 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:03.721374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7591140-151e-45bc-8fb0-56cc1e193d41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:03.741112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1c30a1-f250-4b20-b36d-8aaf7335b9df service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:03.756136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:03.757237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:03.757723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:03.758208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] PATH_INFO: `/v3/users/3f1f545993da432e90090e9cb8fba8b6` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:03.761541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f1f545993da432e90090e9cb8fba8b6)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:03.767553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:03.768968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b7591140-151e-45bc-8fb0-56cc1e193d41 None None] MFA Rules not processed for user `35f28f9458e54fb8a76c740d6f9c33bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:03.772889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b7591140-151e-45bc-8fb0-56cc1e193d41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:03.798633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 886/1763] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:11:03 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:03.891851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f1f545993da432e90090e9cb8fba8b6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:03.909001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f1f545993da432e90090e9cb8fba8b6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:03.932529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f1f545993da432e90090e9cb8fba8b6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:03.967600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f1f545993da432e90090e9cb8fba8b6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:03.969594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-de65e192-e666-458f-a8c7-b59262a53ab8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f1f545993da432e90090e9cb8fba8b6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:03.997144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 878/1764] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:03 2022] DELETE /identity/v3/users/3f1f545993da432e90090e9cb8fba8b6 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:04.017459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7591140-151e-45bc-8fb0-56cc1e193d41 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:04.035590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7591140-151e-45bc-8fb0-56cc1e193d41 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:04.237843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:04.239514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:04.239803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:04.240089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:04.248348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:04.296119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:04.302340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:04.343803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 887/1765] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:04 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:04.612739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:04.613166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:04.613560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:04.664459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] MFA Rules not processed for user `fbdac72f58df4335b3790c5fb1609f5b`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:04.665943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:04.853672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 879/1766] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:04 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:05.985726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:05.999312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:06.036558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:06.039188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:06.170503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:06.224315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e782c01-1e8b-4c38-a5e8-e79c270ad244 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:06.241655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:06.242464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:06.242972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:06.243431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] PATH_INFO: `/v3/projects/2437e7bca09446dbb8febc342d7041fd` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:06.244699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:06.244997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2437e7bca09446dbb8febc342d7041fd)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:06.247376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:06.310349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490dfb96-2717-4509-bf4a-13668e306139 service placement] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:06.324534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:06.325234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:06.325556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:06.325935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] PATH_INFO: `/v3/projects/fd742c3aab8945ffb77d07429bbd2dd4` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:06.327235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd742c3aab8945ffb77d07429bbd2dd4)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:06.329560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:06.377377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2437e7bca09446dbb8febc342d7041fd', 'request_id': 'req-cc403726-0545-4bb2-8f7e-766e3a369f6e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.390525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2437e7bca09446dbb8febc342d7041fd', 'request_id': 'req-cc403726-0545-4bb2-8f7e-766e3a369f6e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.403254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2437e7bca09446dbb8febc342d7041fd', 'request_id': 'req-cc403726-0545-4bb2-8f7e-766e3a369f6e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.420406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] The token cache is being invalidate because project 2437e7bca09446dbb8febc342d7041fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:06.421112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.425129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.431354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 880/1767] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:06 2022] DELETE /identity/v3/projects/2437e7bca09446dbb8febc342d7041fd => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:06.432481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:06.434258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:06.480839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd742c3aab8945ffb77d07429bbd2dd4', 'request_id': 'req-d0916cc7-0a63-4b2f-a561-9185865dfa5a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.505528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd742c3aab8945ffb77d07429bbd2dd4', 'request_id': 'req-d0916cc7-0a63-4b2f-a561-9185865dfa5a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.526681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd742c3aab8945ffb77d07429bbd2dd4', 'request_id': 'req-d0916cc7-0a63-4b2f-a561-9185865dfa5a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.576518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] The token cache is being invalidate because project fd742c3aab8945ffb77d07429bbd2dd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:06.576945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.583175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:06.603536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 888/1768] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:05 2022] DELETE /identity/v3/projects/fd742c3aab8945ffb77d07429bbd2dd4 => generated 0 bytes in 617 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:06.604867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:06.606892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:06.775354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc403726-0545-4bb2-8f7e-766e3a369f6e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:06.985550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0916cc7-0a63-4b2f-a561-9185865dfa5a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:07.003801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:07.004420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:07.004625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:07.004898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] PATH_INFO: `/v3/users/502f968e8bae49d5afb8eb10ca32626c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:07.023223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] RBAC: Authorizing `identity:delete_user(user_id=502f968e8bae49d5afb8eb10ca32626c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:07.037301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:07.047342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:07.048105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:07.048388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:07.053657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] PATH_INFO: `/v3/users/8e94663a93cf40f896d6e84477595930` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:07.055679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e94663a93cf40f896d6e84477595930)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:07.057677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:07.210574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:07.275708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '502f968e8bae49d5afb8eb10ca32626c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.337335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '502f968e8bae49d5afb8eb10ca32626c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.340502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:07.348785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '502f968e8bae49d5afb8eb10ca32626c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.361128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e94663a93cf40f896d6e84477595930'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.374668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '502f968e8bae49d5afb8eb10ca32626c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.375912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-156d59fb-4719-43e8-9997-487147b1b22b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '502f968e8bae49d5afb8eb10ca32626c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.396130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e94663a93cf40f896d6e84477595930'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.413533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 889/1769] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:06 2022] DELETE /identity/v3/users/502f968e8bae49d5afb8eb10ca32626c => generated 0 bytes in 809 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:07.454227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e94663a93cf40f896d6e84477595930'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.499537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e94663a93cf40f896d6e84477595930'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.502541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e94663a93cf40f896d6e84477595930'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:07.537762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 881/1770] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:06 2022] DELETE /identity/v3/users/8e94663a93cf40f896d6e84477595930 => generated 0 bytes in 1104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:07.650851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feffad42-c152-46ad-9c63-0d358d0a9498 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:07.651412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feffad42-c152-46ad-9c63-0d358d0a9498 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:07.651835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feffad42-c152-46ad-9c63-0d358d0a9498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:07.724195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-feffad42-c152-46ad-9c63-0d358d0a9498 None None] MFA Rules not processed for user `9d3fd2b3c6cd47019e15fbf959d47381`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:07.725829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-feffad42-c152-46ad-9c63-0d358d0a9498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:07.864400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 890/1771] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:09.303126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:09.305313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:09.408680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49dbc73-34da-4bb6-989e-0554ae65e819 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:09.409496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49dbc73-34da-4bb6-989e-0554ae65e819 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:09.410375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49dbc73-34da-4bb6-989e-0554ae65e819 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:09.413075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 891/1772] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:11:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:11:09.438963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:09.439292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:09.439735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:09.517547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:09.519852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:09.533967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9bc3b4-e578-4158-a398-aca251cf7f74 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:09.558250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 892/1773] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:11:09 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:09.559975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c8dec5-6d6b-4a97-8adb-d6d640b335c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:09.563475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c8dec5-6d6b-4a97-8adb-d6d640b335c2 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:09.564075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c8dec5-6d6b-4a97-8adb-d6d640b335c2 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:09.564573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c8dec5-6d6b-4a97-8adb-d6d640b335c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:09.583260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c8dec5-6d6b-4a97-8adb-d6d640b335c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:09.587392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c8dec5-6d6b-4a97-8adb-d6d640b335c2 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:09.625723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 882/1774] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:09.744547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:09.746330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:09.766983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a868960-8440-4e34-8bdf-f17b4d3f2904 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:09.775225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:09.775876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:09.776195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f32113-759f-4c23-a594-1c4d3cce6d22 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:09.776319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:09.776513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:09.776746 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f32113-759f-4c23-a594-1c4d3cce6d22 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:09.777123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f32113-759f-4c23-a594-1c4d3cce6d22 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:09.778783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 883/1775] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:11:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:11:09.790178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:09.790541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63fef3-805e-456d-81eb-d110e04ec567 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:09.790935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63fef3-805e-456d-81eb-d110e04ec567 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:09.791341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd63fef3-805e-456d-81eb-d110e04ec567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:09.792115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:09.820272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 893/1776] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:09 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:09.826717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-fd63fef3-805e-456d-81eb-d110e04ec567 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:09.828362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fd63fef3-805e-456d-81eb-d110e04ec567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:09.842410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 884/1777] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:11:09 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:09.887263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:09.888906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:09.960958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd63fef3-805e-456d-81eb-d110e04ec567 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:09.983916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd63fef3-805e-456d-81eb-d110e04ec567 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.017377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.021765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.024219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.024511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.037009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59334748-80a6-4922-b7aa-7822755cdcf6 service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.050670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.058773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.062575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.063422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.063807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.064685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] PATH_INFO: `/v3/projects/e6d19b54094d4429b07087b208d9b78f` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.067146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6d19b54094d4429b07087b208d9b78f)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.092162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.114887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 885/1778] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:09 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:10.115855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:10.117547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:10.144466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cb74cc-d65c-4b2d-8420-8ca4b81ec1fc service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.160607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.161234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.161520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.161804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] PATH_INFO: `/v3/projects/afd263fdeaa345bba964bb0b516787e3` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.163110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=afd263fdeaa345bba964bb0b516787e3)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.165924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.242007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6d19b54094d4429b07087b208d9b78f', 'request_id': 'req-99ca2ae6-91e0-4385-aebf-302f9de38d29'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.258907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6d19b54094d4429b07087b208d9b78f', 'request_id': 'req-99ca2ae6-91e0-4385-aebf-302f9de38d29'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.260556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:10.268880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afd263fdeaa345bba964bb0b516787e3', 'request_id': 'req-9fc409ab-9035-46ed-9383-08e36c6c9e14'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.271377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6d19b54094d4429b07087b208d9b78f', 'request_id': 'req-99ca2ae6-91e0-4385-aebf-302f9de38d29'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.290523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afd263fdeaa345bba964bb0b516787e3', 'request_id': 'req-9fc409ab-9035-46ed-9383-08e36c6c9e14'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.293219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:10.294338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] The token cache is being invalidate because project e6d19b54094d4429b07087b208d9b78f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:10.294849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.296665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99ca2ae6-91e0-4385-aebf-302f9de38d29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.299244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 894/1779] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:09 2022] DELETE /identity/v3/projects/e6d19b54094d4429b07087b208d9b78f => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:10.300919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afd263fdeaa345bba964bb0b516787e3', 'request_id': 'req-9fc409ab-9035-46ed-9383-08e36c6c9e14'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.325751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1269c478-7818-406b-8e55-9aa808ca055c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.325751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1269c478-7818-406b-8e55-9aa808ca055c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.326675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1269c478-7818-406b-8e55-9aa808ca055c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.332357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] The token cache is being invalidate because project afd263fdeaa345bba964bb0b516787e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:10.333273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.338224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9fc409ab-9035-46ed-9383-08e36c6c9e14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:10.342405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 886/1780] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:10 2022] DELETE /identity/v3/projects/afd263fdeaa345bba964bb0b516787e3 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:10.347773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dca9a7-79a3-4aa4-aa3d-52571e0f64fb None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.348165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dca9a7-79a3-4aa4-aa3d-52571e0f64fb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.348631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dca9a7-79a3-4aa4-aa3d-52571e0f64fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.397829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c5dca9a7-79a3-4aa4-aa3d-52571e0f64fb None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:10.408017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c5dca9a7-79a3-4aa4-aa3d-52571e0f64fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:10.409812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1269c478-7818-406b-8e55-9aa808ca055c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:10.413262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1269c478-7818-406b-8e55-9aa808ca055c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:10.525320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 895/1781] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:10.532174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d1ecb-8a74-430a-8e7f-e2e881a2adde None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.533170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d1ecb-8a74-430a-8e7f-e2e881a2adde None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.533664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d1ecb-8a74-430a-8e7f-e2e881a2adde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.553171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 887/1782] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:10.554731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ae924-cb40-4e27-8d15-18869911b7de None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.555046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ae924-cb40-4e27-8d15-18869911b7de None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.555329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ae924-cb40-4e27-8d15-18869911b7de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.575886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f71d1ecb-8a74-430a-8e7f-e2e881a2adde None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:10.578088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f71d1ecb-8a74-430a-8e7f-e2e881a2adde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:10.602252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 896/1783] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:10.608524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71d1ecb-8a74-430a-8e7f-e2e881a2adde None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:10.610708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-695ae924-cb40-4e27-8d15-18869911b7de None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:10.614147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-695ae924-cb40-4e27-8d15-18869911b7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:10.628318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71d1ecb-8a74-430a-8e7f-e2e881a2adde None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.633593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.634717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.635391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.635774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.636188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 888/1784] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:10.637797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:10.638451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.638779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ae924-cb40-4e27-8d15-18869911b7de None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:10.642093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.660966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ae924-cb40-4e27-8d15-18869911b7de None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.683447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.686636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.692387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 897/1785] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:10.692942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.693522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.695673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:10.696146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.704279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.705686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:10.732497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 889/1786] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:10.737168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:10.744267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719fbb7e-0d91-4e3b-b13d-64352e18a7cd admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.761317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5b8ca0-6be3-4c09-b0d2-4089d432e336 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.761570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.762002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.762287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.762562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.765133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.771487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.779726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.780653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.781074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.782244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.783156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.785262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.840154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 898/1787] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/projects => generated 409 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:10.844237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:10.867185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 890/1788] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/projects => generated 391 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:10.868528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:10.975363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c1116f-bd76-4e43-9ab5-995a542f054b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.980719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.981369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.981649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.981996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.982683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.983535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67b7fd9-055b-4eb8-a631-85b480fffca5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:10.984994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:10.990931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:10.991774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:10.992249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:10.992950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:10.994436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:10.997095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:11.048880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 899/1789] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/users => generated 305 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:11.050906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 891/1790] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:10 2022] POST /identity/v3/users => generated 296 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:11.051870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:11.056366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:11.090978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382500f-dc1a-43ff-a107-df4bb963f699 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:11.105933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:11.108351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.109679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.109975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.111883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:11.116737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:11.144779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 892/1791] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:11.149899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:11.150932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a21fb2-7d3c-420e-8202-6d393251b1a8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:11.169241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:11.170019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.170302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.170579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.175203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:11.179453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:11.180788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ade9fff-67f6-4aec-917a-9907eeb75e4a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:11.195194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:11.195745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.196093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.196424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.199371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:11.212183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:11.219341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 900/1792] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:11.237896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:11.277232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 893/1793] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:11.284106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:11.290651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a00c4e-6f0f-437c-84d9-c96b76a19cef admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:11.305867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:11.309350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.309639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.311170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.311336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3752da8-2b2c-4c16-b064-fce8ca189076 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:11.314716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:11.317848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:11.319395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706a6443-930d-4e66-a383-7147e905d0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:11.320429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706a6443-930d-4e66-a383-7147e905d0b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.320942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706a6443-930d-4e66-a383-7147e905d0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.321419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706a6443-930d-4e66-a383-7147e905d0b1 admin admin] PATH_INFO: `/v3/projects/537f0eabc9054dbda73c63d2b48cd20c/users/f0d694ce6ab8435abf2286f99eed0bfb/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.350225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:DELETE=1,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:11.363156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706a6443-930d-4e66-a383-7147e905d0b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=537f0eabc9054dbda73c63d2b48cd20c, user_id=f0d694ce6ab8435abf2286f99eed0bfb, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:11.366594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706a6443-930d-4e66-a383-7147e905d0b1 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:11.382310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 901/1794] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:11.388103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:11.393880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 894/1795] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:11 2022] PUT /identity/v3/projects/537f0eabc9054dbda73c63d2b48cd20c/users/f0d694ce6ab8435abf2286f99eed0bfb/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:11.410994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab33537-9291-4577-9f86-a218db21203c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.411636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab33537-9291-4577-9f86-a218db21203c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.412176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab33537-9291-4577-9f86-a218db21203c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.416984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13c3bc1-a814-48e4-8c41-cac4ea93f56f admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:11.440945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:11.490051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7ab33537-9291-4577-9f86-a218db21203c None None] MFA Rules not processed for user `f0d694ce6ab8435abf2286f99eed0bfb`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:11.492873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7ab33537-9291-4577-9f86-a218db21203c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:11.587436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955c4cab-0244-4ab1-b45f-815d7ded6dd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:11.588339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955c4cab-0244-4ab1-b45f-815d7ded6dd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.589865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955c4cab-0244-4ab1-b45f-815d7ded6dd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.590259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955c4cab-0244-4ab1-b45f-815d7ded6dd0 admin admin] PATH_INFO: `/v3/projects/7b252ac5c51048f5827563ab131df0ff/users/97218f8a89024ae3a7d393fe04dcf416/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.636398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955c4cab-0244-4ab1-b45f-815d7ded6dd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b252ac5c51048f5827563ab131df0ff, user_id=97218f8a89024ae3a7d393fe04dcf416, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:11.642627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955c4cab-0244-4ab1-b45f-815d7ded6dd0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:11.675296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 902/1796] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:11 2022] PUT /identity/v3/projects/7b252ac5c51048f5827563ab131df0ff/users/97218f8a89024ae3a7d393fe04dcf416/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:11.690229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f22fdf-2b93-46e7-8df1-6a3fec6071c8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.690715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f22fdf-2b93-46e7-8df1-6a3fec6071c8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.691179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f22fdf-2b93-46e7-8df1-6a3fec6071c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.729121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 895/1797] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:11 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:11.730377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab33537-9291-4577-9f86-a218db21203c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:11.770084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab33537-9291-4577-9f86-a218db21203c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:11.841997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f5f22fdf-2b93-46e7-8df1-6a3fec6071c8 None None] MFA Rules not processed for user `97218f8a89024ae3a7d393fe04dcf416`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:11.854401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f5f22fdf-2b93-46e7-8df1-6a3fec6071c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:11.899923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:11.900662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:11.901011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:11.901485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:11.902174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:11.906019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.017508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 896/1798] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:11 2022] POST /identity/v3/projects => generated 405 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:12.022189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.043982 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 903/1799] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:11 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 355 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:12.048456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f22fdf-2b93-46e7-8df1-6a3fec6071c8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.084017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f22fdf-2b93-46e7-8df1-6a3fec6071c8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.136964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5367ea67-f1f4-4a51-85ea-3a2c030ac9cd admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.142329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.143205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.143670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.144247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.145015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.147679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.169190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:12.182976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.183984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.184271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.184986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.185949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.188685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.192866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 897/1800] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:12 2022] POST /identity/v3/users => generated 309 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:12.196717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.213143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c911fa-9e6b-43b7-973d-48b4ad98c374 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.223971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.224959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.225789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.226473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.228709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.231220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 904/1801] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:12 2022] POST /identity/v3/projects => generated 393 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:12.232521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.235269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.263582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:12.273420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 898/1802] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:12.276794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.303466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b55a3-f3fe-46f6-af05-2488e252be4c admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.404512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05aef09b-87c6-4989-b949-2e97ce88ac06 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.410843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.411695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.412071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.412412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.413149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.415409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.459273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 905/1803] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:12 2022] POST /identity/v3/users => generated 303 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:12.463271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.476493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.477217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.477500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.477794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] PATH_INFO: `/v3/projects/5118f77ad2ec48448339a9c70d6fa099/users/7962410dcb784b0c97c67df9da0025c2/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.499342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b1f130-500f-41a9-b870-6a676025db58 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.512972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.513856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.514310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.514708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.515535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.517504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.518983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5118f77ad2ec48448339a9c70d6fa099, user_id=7962410dcb784b0c97c67df9da0025c2, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.524681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.532866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 906/1804] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:12.539346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.558895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 899/1805] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:12 2022] PUT /identity/v3/projects/5118f77ad2ec48448339a9c70d6fa099/users/7962410dcb784b0c97c67df9da0025c2/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:12.559125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6898aba-1880-44e7-973a-a3fce5654c50 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.560270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.648952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.649213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.649498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.649773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] PATH_INFO: `/v3/projects/4ad2b52fa7424a4ebfb8c802cf1c8488/users/a6d6d919aa9044aabb0f5dc552c68624/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.674338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ad2b52fa7424a4ebfb8c802cf1c8488, user_id=a6d6d919aa9044aabb0f5dc552c68624, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.678676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.717665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 907/1806] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:12 2022] PUT /identity/v3/projects/4ad2b52fa7424a4ebfb8c802cf1c8488/users/a6d6d919aa9044aabb0f5dc552c68624/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:12.719618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.730293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3040646f-2167-4de9-abfb-ff3887faed35 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.883109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f515e7f3-dc13-4f3d-8f7f-a478f885377e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.893333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.894398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.895039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.895707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.896810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.897389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.898046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.898321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.898589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.899331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.900946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.902207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.921435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 900/1807] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:12.928339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.928990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 908/1808] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:12 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:12.933276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:12.948214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0296636-9bc0-4ea5-a682-7de80b67f62f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.955981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a04e0b-58a3-4c8b-bd71-47696a462933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.957127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a04e0b-58a3-4c8b-bd71-47696a462933 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.957622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a04e0b-58a3-4c8b-bd71-47696a462933 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.958336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a04e0b-58a3-4c8b-bd71-47696a462933 admin admin] PATH_INFO: `/v3/projects/5118f77ad2ec48448339a9c70d6fa099/users/7962410dcb784b0c97c67df9da0025c2/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.961101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a04e0b-58a3-4c8b-bd71-47696a462933 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5118f77ad2ec48448339a9c70d6fa099, user_id=7962410dcb784b0c97c67df9da0025c2, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.963585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52933345-0f20-4b12-9054-06ae58a4eae1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:12.965928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a04e0b-58a3-4c8b-bd71-47696a462933 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.974154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:12.976833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b9e10e-5a93-450e-bad2-89c29738afb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:12.977655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b9e10e-5a93-450e-bad2-89c29738afb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:12.977869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b9e10e-5a93-450e-bad2-89c29738afb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:12.978417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b9e10e-5a93-450e-bad2-89c29738afb6 admin admin] PATH_INFO: `/v3/projects/4ad2b52fa7424a4ebfb8c802cf1c8488/users/a6d6d919aa9044aabb0f5dc552c68624/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:12.989890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b9e10e-5a93-450e-bad2-89c29738afb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ad2b52fa7424a4ebfb8c802cf1c8488, user_id=a6d6d919aa9044aabb0f5dc552c68624, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:12.992682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b9e10e-5a93-450e-bad2-89c29738afb6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:12.993967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 901/1809] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:12 2022] PUT /identity/v3/projects/5118f77ad2ec48448339a9c70d6fa099/users/7962410dcb784b0c97c67df9da0025c2/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:13.003853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:13.006736 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:13.008767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:13.009946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:13.026578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 909/1810] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:12 2022] PUT /identity/v3/projects/4ad2b52fa7424a4ebfb8c802cf1c8488/users/a6d6d919aa9044aabb0f5dc552c68624/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:13.037126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:13.037480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:13.039156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:13.076073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] MFA Rules not processed for user `7962410dcb784b0c97c67df9da0025c2`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:13.079012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:13.107735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] MFA Rules not processed for user `a6d6d919aa9044aabb0f5dc552c68624`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:13.111765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:13.274655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 902/1811] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:13 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:13.295481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:13.297073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:13.315932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 910/1812] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:13 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:13.527455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7bc235-aa1f-4457-bf71-bfed3fcb148a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:13.541861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:13.544466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:13.544888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:13.545800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:13.564261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:13.566269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:13.586769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 903/1813] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:13 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:13.638320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:13.640089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:13.661094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f4b6a8-bb88-402d-9fce-5c46a852d475 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:13.666641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:13.667769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:13.668150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:13.668723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:13.686367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:13.688759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:13.712510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 911/1814] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:13 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:13.816635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:13.834607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5074fba4-3a2a-44e2-a71e-504a7f6fa7f9 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:13.840419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e783ae-cc71-4925-a2a1-ab807e4e2e24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:13.842050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e783ae-cc71-4925-a2a1-ab807e4e2e24 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:13.842769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e783ae-cc71-4925-a2a1-ab807e4e2e24 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:13.843309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e783ae-cc71-4925-a2a1-ab807e4e2e24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:13.936832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:13.948487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cde483-0762-404a-85b4-bca373446173 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:13.958867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e783ae-cc71-4925-a2a1-ab807e4e2e24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:13.960502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e783ae-cc71-4925-a2a1-ab807e4e2e24 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:13.962990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:13.963753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:13.964146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:13.964514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:13.980913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 904/1815] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:13 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:14.151487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:14.153202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:14.177070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 912/1816] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:13 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:14.931210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9537714-677d-4147-9dc2-614de1d04569 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:14.931763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9537714-677d-4147-9dc2-614de1d04569 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:14.931850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9537714-677d-4147-9dc2-614de1d04569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:14.952231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a9537714-677d-4147-9dc2-614de1d04569 None None] MFA Rules not processed for user `9d3fd2b3c6cd47019e15fbf959d47381`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:14.954049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a9537714-677d-4147-9dc2-614de1d04569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:14.972506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:15.022624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 905/1817] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:22.428395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:22.523021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4277c26c-d49d-439b-9fe6-0d87f57c86d9 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:22.529451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:22.530395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:22.530736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:22.531115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] PATH_INFO: `/v3/users/97218f8a89024ae3a7d393fe04dcf416` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:22.532380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] RBAC: Authorizing `identity:delete_user(user_id=97218f8a89024ae3a7d393fe04dcf416)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:22.534935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:22.554769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:22.626406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97218f8a89024ae3a7d393fe04dcf416'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:22.640264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97218f8a89024ae3a7d393fe04dcf416'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:22.655235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97218f8a89024ae3a7d393fe04dcf416'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:22.667428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97218f8a89024ae3a7d393fe04dcf416'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:22.668196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97218f8a89024ae3a7d393fe04dcf416'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:22.680342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 913/1818] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:22 2022] DELETE /identity/v3/users/97218f8a89024ae3a7d393fe04dcf416 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:22.682532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:22.683059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:22.683153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:22.721942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:22.723326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:22.818258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 906/1819] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:22.874057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:22.878149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:22.939452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e5af1-77aa-4961-a5e5-99ca8447fc0a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:22.943587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e07f38-9ff0-4fda-adec-be6b8d5223f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:22.944955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e07f38-9ff0-4fda-adec-be6b8d5223f3 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:22.945242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e07f38-9ff0-4fda-adec-be6b8d5223f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:22.945558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e07f38-9ff0-4fda-adec-be6b8d5223f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:22.956997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e07f38-9ff0-4fda-adec-be6b8d5223f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:22.958572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e07f38-9ff0-4fda-adec-be6b8d5223f3 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:22.993156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 914/1820] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:23.094502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:23.096350 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:23.128902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e696f55-52d6-4b4b-8848-f3df01f0f4d6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:23.155416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:23.156437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:23.156766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:23.157064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] PATH_INFO: `/v3/users/acc5e42ec3e9424db26987a312f66a44` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:23.158577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=acc5e42ec3e9424db26987a312f66a44)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:23.160976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:23.231188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'acc5e42ec3e9424db26987a312f66a44'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:23.241000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'acc5e42ec3e9424db26987a312f66a44'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:23.254307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'acc5e42ec3e9424db26987a312f66a44'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:23.275375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'acc5e42ec3e9424db26987a312f66a44'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:23.276205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'acc5e42ec3e9424db26987a312f66a44'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:23.299353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 907/1821] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:23 2022] DELETE /identity/v3/users/acc5e42ec3e9424db26987a312f66a44 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:23.304583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2469796-934c-4966-a0fb-542d2c3bbb25 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:23.305255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2469796-934c-4966-a0fb-542d2c3bbb25 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:23.305652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2469796-934c-4966-a0fb-542d2c3bbb25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:23.326199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d2469796-934c-4966-a0fb-542d2c3bbb25 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:23.328998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d2469796-934c-4966-a0fb-542d2c3bbb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:23.421491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 915/1822] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:23.435952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:23.438042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:23.525605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:23.549092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42175c-763c-41a2-a4d7-c09b61d7bc7e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:23.558440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea785e57-c377-4cbc-8274-abe7947d59ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:23.559605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea785e57-c377-4cbc-8274-abe7947d59ad service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:23.560078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea785e57-c377-4cbc-8274-abe7947d59ad service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:23.560586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea785e57-c377-4cbc-8274-abe7947d59ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:23.580596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea785e57-c377-4cbc-8274-abe7947d59ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:23.582383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea785e57-c377-4cbc-8274-abe7947d59ad service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:23.611337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 908/1823] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:23.788273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d045e-a844-4b47-8403-cdfe08008e5b None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:23.789186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d045e-a844-4b47-8403-cdfe08008e5b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:23.789857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8d045e-a844-4b47-8403-cdfe08008e5b None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:23.791929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 916/1824] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:11:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:11:23.804412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:23.804878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:23.805472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:23.851795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:23.853613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:23.936563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 909/1825] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:11:23 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:23.952259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8d045e-a844-4b47-8403-cdfe08008e5b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:23.954511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6e8d045e-a844-4b47-8403-cdfe08008e5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:23.978423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8d045e-a844-4b47-8403-cdfe08008e5b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:23.999401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:24.000067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:24.003937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:24.004309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:24.035762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:24.039404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:24.066540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 917/1826] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:23 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:24.160907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:24.163139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:24.185497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2410dd52-ee0e-4f50-a0e2-1a2cad52cbe9 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:24.199123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:24.200258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:24.200525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:24.200917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] PATH_INFO: `/v3/projects/7b252ac5c51048f5827563ab131df0ff` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:24.205142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b252ac5c51048f5827563ab131df0ff)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:24.209374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:24.314748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b252ac5c51048f5827563ab131df0ff', 'request_id': 'req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.335305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b252ac5c51048f5827563ab131df0ff', 'request_id': 'req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.345954 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b252ac5c51048f5827563ab131df0ff', 'request_id': 'req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.379523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] The token cache is being invalidate because project 7b252ac5c51048f5827563ab131df0ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:24.381090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.382614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.386382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 910/1827] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:24 2022] DELETE /identity/v3/projects/7b252ac5c51048f5827563ab131df0ff => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:24.391292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:24.408006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c867bb-6058-43bc-b09e-d4762580094a service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:24.435301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:24.494436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:24.495309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:24.495804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:24.496180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] PATH_INFO: `/v3/users/a6d6d919aa9044aabb0f5dc552c68624` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:24.497470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6d6d919aa9044aabb0f5dc552c68624)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:24.499755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:24.570696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6d6d919aa9044aabb0f5dc552c68624'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.586794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6d6d919aa9044aabb0f5dc552c68624'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.598019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6d6d919aa9044aabb0f5dc552c68624'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.611868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6d6d919aa9044aabb0f5dc552c68624'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.613505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6d6d919aa9044aabb0f5dc552c68624'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:24.626687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 918/1828] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:24 2022] DELETE /identity/v3/users/a6d6d919aa9044aabb0f5dc552c68624 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:25.195276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:25.197268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.225829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b33fb-3e49-4c09-bde0-f2d71ed8beeb admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:25.357348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:25.367586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:25.369597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.370633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.372457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] PATH_INFO: `/v3/projects/3582316ee9b9472a9fda807af4b25c62` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.374392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] RBAC: Authorizing `identity:delete_project(project_id=3582316ee9b9472a9fda807af4b25c62)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:25.380215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:25.431396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:25.433731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.458581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3582316ee9b9472a9fda807af4b25c62', 'request_id': 'req-b79abd4c-af91-42f9-8b17-c390aeab156a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.460274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11bb410-018f-476a-b24a-56b22f4e9d06 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:25.468888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3582316ee9b9472a9fda807af4b25c62', 'request_id': 'req-b79abd4c-af91-42f9-8b17-c390aeab156a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.476477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3582316ee9b9472a9fda807af4b25c62', 'request_id': 'req-b79abd4c-af91-42f9-8b17-c390aeab156a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.493296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] The token cache is being invalidate because project 3582316ee9b9472a9fda807af4b25c62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:25.493661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.494378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.496857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 911/1829] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:25 2022] DELETE /identity/v3/projects/3582316ee9b9472a9fda807af4b25c62 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:25.497827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:25.500146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.542559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:25.543266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.543554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.543830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] PATH_INFO: `/v3/projects/4ad2b52fa7424a4ebfb8c802cf1c8488` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.545000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ad2b52fa7424a4ebfb8c802cf1c8488)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:25.547003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:25.568540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79abd4c-af91-42f9-8b17-c390aeab156a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:25.573606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:25.574390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.574764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.575158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] PATH_INFO: `/v3/users/f0d694ce6ab8435abf2286f99eed0bfb` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.576490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0d694ce6ab8435abf2286f99eed0bfb)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:25.578873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:25.591526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:25.605356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ad2b52fa7424a4ebfb8c802cf1c8488', 'request_id': 'req-69befbb3-0d10-4d56-a06e-a0d8a8479439'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.612701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ad2b52fa7424a4ebfb8c802cf1c8488', 'request_id': 'req-69befbb3-0d10-4d56-a06e-a0d8a8479439'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.619159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ad2b52fa7424a4ebfb8c802cf1c8488', 'request_id': 'req-69befbb3-0d10-4d56-a06e-a0d8a8479439'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.629900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0d694ce6ab8435abf2286f99eed0bfb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.630661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] The token cache is being invalidate because project 4ad2b52fa7424a4ebfb8c802cf1c8488 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:25.631032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.631615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.633352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 919/1830] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:25 2022] DELETE /identity/v3/projects/4ad2b52fa7424a4ebfb8c802cf1c8488 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:25.633903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:25.635198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.639344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0d694ce6ab8435abf2286f99eed0bfb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.647907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0d694ce6ab8435abf2286f99eed0bfb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.660344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0d694ce6ab8435abf2286f99eed0bfb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.661016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5e387d91-d95a-4954-9b70-a587cea904e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0d694ce6ab8435abf2286f99eed0bfb'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.667041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 912/1831] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:25 2022] DELETE /identity/v3/users/f0d694ce6ab8435abf2286f99eed0bfb => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:25.668321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39205f-69b2-4d26-afae-4022a237841d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.668601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39205f-69b2-4d26-afae-4022a237841d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.668869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39205f-69b2-4d26-afae-4022a237841d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.682190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:25.688427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3d39205f-69b2-4d26-afae-4022a237841d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:25.690747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3d39205f-69b2-4d26-afae-4022a237841d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.702897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69befbb3-0d10-4d56-a06e-a0d8a8479439 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:25.764004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:25.764777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 913/1832] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:25.766174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e7ba34-5bc8-4b1a-a570-cf2fae163e55 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.766461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e7ba34-5bc8-4b1a-a570-cf2fae163e55 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.766742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e7ba34-5bc8-4b1a-a570-cf2fae163e55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.770984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.771300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.771579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] PATH_INFO: `/v3/users/e7193ed031a54d6bbe2c1c174d9ef997` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.772925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7193ed031a54d6bbe2c1c174d9ef997)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:25.774922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:25.790665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f2e7ba34-5bc8-4b1a-a570-cf2fae163e55 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:25.792898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f2e7ba34-5bc8-4b1a-a570-cf2fae163e55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.804083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 914/1833] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:25.806404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.812864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.813232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.835621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7193ed031a54d6bbe2c1c174d9ef997'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.836086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:25.837730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.847892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 915/1834] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:25.848980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:25.852162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7193ed031a54d6bbe2c1c174d9ef997'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.853917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.858400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:25.860727 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7193ed031a54d6bbe2c1c174d9ef997'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.871300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7193ed031a54d6bbe2c1c174d9ef997'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.872194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7193ed031a54d6bbe2c1c174d9ef997'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:25.880719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 920/1835] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:25 2022] DELETE /identity/v3/users/e7193ed031a54d6bbe2c1c174d9ef997 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:25.881599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:25.883422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:25.932616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ba9fa-09b3-47ea-8da9-0feaa444d2b1 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:25.977783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd66d10c-664d-49a0-a04c-c9a8126c3ecb admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:25.983438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:25.984121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:25.984477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:25.984827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:25.985648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:25.986022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:25.987935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:25.999699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.000390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.000824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.001153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.003972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:26.009493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 921/1836] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:26.013016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.026041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.027133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29847d05-0865-420f-95cd-5d9a893ef9a9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.027607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.035272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.036035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.036329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.036712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.037686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.040002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.041459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 916/1837] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:26.078938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 922/1838] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:26 2022] POST /identity/v3/projects => generated 367 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:26.080458 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.090646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab335a76-fc4a-4bc6-be75-f054fd7f6be8 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.151190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:26.164341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.165427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.165849 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.166268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.167204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.169508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.206734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 917/1839] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:26 2022] POST /identity/v3/users => generated 284 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:26.208028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.219161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9b341-b4e3-4c2f-9421-6d4704c8cc53 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.225143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.225465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.225659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.225925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.226647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.228320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.239947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 923/1840] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:26.244172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.256521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36a0a38-0e17-41b8-a54c-d8ee00611f18 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.262034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc33f6fc-5f93-4727-8d8e-ce16058cebd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.262834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33f6fc-5f93-4727-8d8e-ce16058cebd5 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.263252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33f6fc-5f93-4727-8d8e-ce16058cebd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.263636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33f6fc-5f93-4727-8d8e-ce16058cebd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.264639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc33f6fc-5f93-4727-8d8e-ce16058cebd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.266642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc33f6fc-5f93-4727-8d8e-ce16058cebd5 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.281720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 918/1841] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:26 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:26.283090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.294547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0243d989-3e7a-4f5c-b8f9-bb8317495bd5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.300289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.302215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.302215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.302215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] PATH_INFO: `/v3/projects/cdee78f3a79e4e7dab0cedd5aa21cac6/users/273dff55312c42b7a94316e4d83af9cd/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.319408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cdee78f3a79e4e7dab0cedd5aa21cac6, user_id=273dff55312c42b7a94316e4d83af9cd, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.322253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.343476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 924/1842] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:26 2022] PUT /identity/v3/projects/cdee78f3a79e4e7dab0cedd5aa21cac6/users/273dff55312c42b7a94316e4d83af9cd/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:26.355574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.356308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.356810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.389388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] MFA Rules not processed for user `273dff55312c42b7a94316e4d83af9cd`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:26.390785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:26.478127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 919/1843] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:26 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:26.479414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.612467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613e7206-63b3-4fa5-9184-878995367f5e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.621836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.623162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.624219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.624219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.627053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.629940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.674347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.676081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:26.712925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34301ec5-2172-43af-b18e-4d04ef7b29d5 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.714010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 925/1844] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:26 2022] POST /identity/v3/projects => generated 365 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:26.718718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.748437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:26.839630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792ee42b-c5bd-49f7-9ca8-d17d8a875d51 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.845875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.846554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.846839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.847160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] PATH_INFO: `/v3/projects/4e81ec3b6314454c9201ab5d935d24fd` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.848737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e81ec3b6314454c9201ab5d935d24fd)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.850679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.854145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.862394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.862801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.863274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.863997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.866045 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.902193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 926/1845] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:26 2022] POST /identity/v3/users => generated 289 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:26.903325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.915014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38cf57-0fc9-4245-a0ac-c07380c73134 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:26.929234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:26.929234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:26.929234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:26.933033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:26.934139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:26.941349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:26.966607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 927/1846] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:26 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:26.967607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:26.979416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27025fb3-551d-4b8f-a516-0a421d449310 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.001411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e81ec3b6314454c9201ab5d935d24fd', 'request_id': 'req-8896b33e-4147-4a1e-a222-3f4d12f569af'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.046253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e81ec3b6314454c9201ab5d935d24fd', 'request_id': 'req-8896b33e-4147-4a1e-a222-3f4d12f569af'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.046253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:27.046253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e81ec3b6314454c9201ab5d935d24fd', 'request_id': 'req-8896b33e-4147-4a1e-a222-3f4d12f569af'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.069984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] The token cache is being invalidate because project 4e81ec3b6314454c9201ab5d935d24fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:27.070266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.074132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.077234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 920/1847] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:26 2022] DELETE /identity/v3/projects/4e81ec3b6314454c9201ab5d935d24fd => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:27.078552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.082552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.090898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd7a8bb-42de-4015-9993-bfc8a2ac1cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.092145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd7a8bb-42de-4015-9993-bfc8a2ac1cc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.092577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd7a8bb-42de-4015-9993-bfc8a2ac1cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.093006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd7a8bb-42de-4015-9993-bfc8a2ac1cc6 admin admin] PATH_INFO: `/v3/projects/d3fe38e8d7fa4639aa60738319c61177/users/d65864b791a344a3af132ad56a276bfa/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.108442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8896b33e-4147-4a1e-a222-3f4d12f569af admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.111077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd7a8bb-42de-4015-9993-bfc8a2ac1cc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3fe38e8d7fa4639aa60738319c61177, user_id=d65864b791a344a3af132ad56a276bfa, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.120991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd7a8bb-42de-4015-9993-bfc8a2ac1cc6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.123632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.124722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.124722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.124924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] PATH_INFO: `/v3/projects/537f0eabc9054dbda73c63d2b48cd20c` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.126262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=537f0eabc9054dbda73c63d2b48cd20c)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.128532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.171808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 928/1848] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:26 2022] PUT /identity/v3/projects/d3fe38e8d7fa4639aa60738319c61177/users/d65864b791a344a3af132ad56a276bfa/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:27.173316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.173605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.173883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.191662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '537f0eabc9054dbda73c63d2b48cd20c', 'request_id': 'req-c50329f4-3b32-402a-8daa-a23a14f918b8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.198917 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:27.200614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.202396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '537f0eabc9054dbda73c63d2b48cd20c', 'request_id': 'req-c50329f4-3b32-402a-8daa-a23a14f918b8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.214001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '537f0eabc9054dbda73c63d2b48cd20c', 'request_id': 'req-c50329f4-3b32-402a-8daa-a23a14f918b8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.226799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] The token cache is being invalidate because project 537f0eabc9054dbda73c63d2b48cd20c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:27.227138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.227891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.229737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 921/1849] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:27 2022] DELETE /identity/v3/projects/537f0eabc9054dbda73c63d2b48cd20c => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:27.230294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.231739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.275835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 929/1850] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:27.276833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.278248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.292483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b932f80-a7fd-4391-8acc-946b630cd78d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.298397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.299344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.299732 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.300121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] PATH_INFO: `/v3/users/7962410dcb784b0c97c67df9da0025c2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.301443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7962410dcb784b0c97c67df9da0025c2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.303639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.325941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:27.343973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50329f4-3b32-402a-8daa-a23a14f918b8 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.349123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d2bd28-4201-46f0-bbe5-20873d72bab9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.349782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2bd28-4201-46f0-bbe5-20873d72bab9 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.350065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2bd28-4201-46f0-bbe5-20873d72bab9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.350342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d2bd28-4201-46f0-bbe5-20873d72bab9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.351133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d2bd28-4201-46f0-bbe5-20873d72bab9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.353017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d2bd28-4201-46f0-bbe5-20873d72bab9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.367667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 922/1851] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:27 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:27.369078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c5c71-846b-457d-a444-96145fa0ef5e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.369368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c5c71-846b-457d-a444-96145fa0ef5e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.369648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c5c71-846b-457d-a444-96145fa0ef5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.397111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-430c5c71-846b-457d-a444-96145fa0ef5e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:27.398360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-430c5c71-846b-457d-a444-96145fa0ef5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.404886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7962410dcb784b0c97c67df9da0025c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.410846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 923/1852] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:27.411821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430c5c71-846b-457d-a444-96145fa0ef5e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.422680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430c5c71-846b-457d-a444-96145fa0ef5e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.427662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7962410dcb784b0c97c67df9da0025c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.429217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.430108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.430450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.431046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] PATH_INFO: `/v3/projects/d3fe38e8d7fa4639aa60738319c61177/users/d65864b791a344a3af132ad56a276bfa/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.433683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3fe38e8d7fa4639aa60738319c61177, user_id=d65864b791a344a3af132ad56a276bfa, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.436912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.440944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7962410dcb784b0c97c67df9da0025c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.452664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 924/1853] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:27 2022] PUT /identity/v3/projects/d3fe38e8d7fa4639aa60738319c61177/users/d65864b791a344a3af132ad56a276bfa/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:27.453855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.462657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7962410dcb784b0c97c67df9da0025c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.465767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-128a1bde-f1b8-472b-b282-cb4bd2efadd4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7962410dcb784b0c97c67df9da0025c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:27.479766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 930/1854] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:27 2022] DELETE /identity/v3/users/7962410dcb784b0c97c67df9da0025c2 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:27.481459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.481882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.482163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.519266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] MFA Rules not processed for user `d65864b791a344a3af132ad56a276bfa`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:27.521178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.571603 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:27.588853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66667144-2609-4582-bfd1-b8615c41d958 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.682522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 931/1855] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:27 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:27.695233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.696677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.768385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.769058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.769446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.769728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.770895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:27.771249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.773080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.805565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9964f08-9926-4ac3-91d0-b39be6852312 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.818393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.819385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.819812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.820248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.831256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 925/1856] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 376 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:27.832278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.836924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:27.838669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.840330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.867992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 932/1857] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:27 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:27.888021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d3d951-4bfc-4bc9-838f-3abcbf1f9421 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:27.903492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:27.904277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:27.904930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:27.905294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:27.906268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:27.908476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:27.955691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 926/1858] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:27 2022] POST /identity/v3/projects => generated 383 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:27.959774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:27.991178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e557b018-6fba-4219-923a-f1ccaf7b9205 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.020542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:28.054291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:28.153202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.154225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.156368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.156812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.157680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.160005 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.161231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af02a5a1-88ed-4132-9575-89a9cf3763d2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.183142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.183847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.184214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.184386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.188927 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:28.195971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 933/1859] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:27 2022] POST /identity/v3/users => generated 292 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:28.201240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:28.210024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:28.224852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cf267b-bee1-4f71-89f0-4cad9f2ea230 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.230408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.231106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.231389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.231685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] PATH_INFO: `/v3/projects/5118f77ad2ec48448339a9c70d6fa099` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.233378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5118f77ad2ec48448339a9c70d6fa099)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.235354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.280491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.281958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.308997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 927/1860] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:28 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:28.311815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:28.323600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5118f77ad2ec48448339a9c70d6fa099', 'request_id': 'req-3ec41c58-3ddd-47fd-a7e8-606d9669522f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:28.345271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5118f77ad2ec48448339a9c70d6fa099', 'request_id': 'req-3ec41c58-3ddd-47fd-a7e8-606d9669522f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:28.355490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5118f77ad2ec48448339a9c70d6fa099', 'request_id': 'req-3ec41c58-3ddd-47fd-a7e8-606d9669522f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:28.360023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67022f07-cbc8-4353-ac7b-a2512ac5e9f4 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.386560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] The token cache is being invalidate because project 5118f77ad2ec48448339a9c70d6fa099 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:28.386848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:28.402537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3ec41c58-3ddd-47fd-a7e8-606d9669522f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:28.409266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:28.412101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 934/1861] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:28 2022] DELETE /identity/v3/projects/5118f77ad2ec48448339a9c70d6fa099 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:28.418981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049782f-f234-4e97-b571-ca09cc592bf9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.419282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049782f-f234-4e97-b571-ca09cc592bf9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.421520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c049782f-f234-4e97-b571-ca09cc592bf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.459233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c049782f-f234-4e97-b571-ca09cc592bf9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:28.460649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c049782f-f234-4e97-b571-ca09cc592bf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:28.473219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.476583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.476865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.477257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.478324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.481370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.502993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 928/1862] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:28 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:28.506756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:28.513591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:28.552111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576e5f8-13c9-41bc-97c2-a2e3cce2536f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.556740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 935/1863] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:28.566213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.566995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc40c3-266e-416a-9435-d2796eae20c0 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.567243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc40c3-266e-416a-9435-d2796eae20c0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.567497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc40c3-266e-416a-9435-d2796eae20c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.569409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.569863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.570289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.571366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.576115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.603156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 929/1864] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:28 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:28.607221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:28.611484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-13cc40c3-266e-416a-9435-d2796eae20c0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:28.616156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-13cc40c3-266e-416a-9435-d2796eae20c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:28.636241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 936/1865] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:28.638932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cc40c3-266e-416a-9435-d2796eae20c0 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:28.639645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5374a7-e6bb-4edc-96d8-d95bc259eee9 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.653529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb8a5fa-11a1-445a-802a-10a5862da298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.656437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb8a5fa-11a1-445a-802a-10a5862da298 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.657727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb8a5fa-11a1-445a-802a-10a5862da298 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.658269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb8a5fa-11a1-445a-802a-10a5862da298 admin admin] PATH_INFO: `/v3/projects/62dde54a654a4239a3f568da765afdad/users/8dbce29803de469cbbcc8a585fea9528/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.664940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cc40c3-266e-416a-9435-d2796eae20c0 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.681022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.682499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.682791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.683102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.683833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:28.684161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.689298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.697229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb8a5fa-11a1-445a-802a-10a5862da298 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62dde54a654a4239a3f568da765afdad, user_id=8dbce29803de469cbbcc8a585fea9528, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.701050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb8a5fa-11a1-445a-802a-10a5862da298 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.713971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 937/1866] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:28.718527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:28.727509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 930/1867] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:28 2022] PUT /identity/v3/projects/62dde54a654a4239a3f568da765afdad/users/8dbce29803de469cbbcc8a585fea9528/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:28.731560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306c31e-6b0a-44a0-b0fd-62fee4e6575e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:28.738140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2759352-51b5-4c19-946e-731d7a9ea2f1 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.738804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2759352-51b5-4c19-946e-731d7a9ea2f1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.739409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2759352-51b5-4c19-946e-731d7a9ea2f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.741516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:28.831208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d2759352-51b5-4c19-946e-731d7a9ea2f1 None None] MFA Rules not processed for user `8dbce29803de469cbbcc8a585fea9528`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:28.836504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d2759352-51b5-4c19-946e-731d7a9ea2f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:28.887038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:28.890203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:28.891130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:28.891576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:28.897907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:28.902203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:28.987385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 938/1868] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:28 2022] POST /identity/v3/projects => generated 383 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:28.987385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.025121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 931/1869] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:29.030531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2759352-51b5-4c19-946e-731d7a9ea2f1 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.055559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2759352-51b5-4c19-946e-731d7a9ea2f1 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.100560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d091ae76-4ec5-4923-bacd-15f0d54bae72 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.105309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.106009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.106362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.106741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.107449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.109494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.127561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.128378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.128673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.128934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.129665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.131855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.148175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 939/1870] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:28 2022] POST /identity/v3/users => generated 292 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:29.152142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.163329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:29.166904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 932/1871] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:29 2022] POST /identity/v3/projects => generated 381 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:29.170706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.195143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01b51c6-7098-4459-8725-c852bf7ba0f1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.263134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b351e37-ee6e-484f-80bc-00d2e55fb23a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.270494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.271497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.271880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.272226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.273001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.275215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.290784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.291508 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.292098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.292352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.293297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.295293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.309806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 940/1872] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:29 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:29.311071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 933/1873] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:29 2022] POST /identity/v3/users => generated 297 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:29.315171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.315306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.326198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8233035c-346e-4f55-a368-02da4c2a832a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.326410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c91044-5d93-4e8a-abaa-51a35bdf0f54 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.331303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.331726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.331846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.332124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.332394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.332487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.332833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.333097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.333190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.333927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.334696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.336465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.348119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 934/1874] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:29.348278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 941/1875] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:29.352039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.354984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.363017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ad87d5-4ac9-41c0-9aa5-2059436011a3 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.366926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2b8001-c789-4d77-bd72-059428044bce admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.368044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.368790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.369136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.369473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] PATH_INFO: `/v3/projects/367100e46b2547938289a8006735bc4c/users/205344a849bc42d19565c4aef9647b97/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.373326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a7ce27-0138-442c-9819-4c3b67a6b5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.374080 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7ce27-0138-442c-9819-4c3b67a6b5d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.378638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7ce27-0138-442c-9819-4c3b67a6b5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.379022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a7ce27-0138-442c-9819-4c3b67a6b5d3 admin admin] PATH_INFO: `/v3/projects/5a2f835012a944878682c8eecd04d0be/users/44296dff12d34fcfa8b6a42a598a6e05/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.381898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:29.392303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] RBAC: Authorizing `identity:create_grant(project_id=367100e46b2547938289a8006735bc4c, user_id=205344a849bc42d19565c4aef9647b97, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.396519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.397813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a7ce27-0138-442c-9819-4c3b67a6b5d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a2f835012a944878682c8eecd04d0be, user_id=44296dff12d34fcfa8b6a42a598a6e05, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.401357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a7ce27-0138-442c-9819-4c3b67a6b5d3 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.417776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 935/1876] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:29 2022] PUT /identity/v3/projects/367100e46b2547938289a8006735bc4c/users/205344a849bc42d19565c4aef9647b97/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:29.422015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.424661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 942/1877] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:29 2022] PUT /identity/v3/projects/5a2f835012a944878682c8eecd04d0be/users/44296dff12d34fcfa8b6a42a598a6e05/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:29.434642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.434927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.435309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.479940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] MFA Rules not processed for user `44296dff12d34fcfa8b6a42a598a6e05`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:29.482880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:29.506209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18617957-5169-40f9-a6a6-5abacc886044 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.567896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 943/1878] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:29.581573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.584455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:29.586092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.586864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.587271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.587641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.588451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.590462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.593005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:29.609084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 936/1879] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:29 2022] GET /identity/v3/roles => generated 1420 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:29.611081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.634355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa297a6-38c4-47e1-b288-1b3524d1306b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.639708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc03bdb4-5989-45ce-bd24-76123e4b0c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.642640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03bdb4-5989-45ce-bd24-76123e4b0c38 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.642990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03bdb4-5989-45ce-bd24-76123e4b0c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.643229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03bdb4-5989-45ce-bd24-76123e4b0c38 admin admin] PATH_INFO: `/v3/projects/367100e46b2547938289a8006735bc4c/users/205344a849bc42d19565c4aef9647b97/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.648595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc03bdb4-5989-45ce-bd24-76123e4b0c38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=367100e46b2547938289a8006735bc4c, user_id=205344a849bc42d19565c4aef9647b97, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.651704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc03bdb4-5989-45ce-bd24-76123e4b0c38 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.674339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e80c3b-7800-4473-a30e-e0da1c674154 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.678915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 937/1880] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:29 2022] PUT /identity/v3/projects/367100e46b2547938289a8006735bc4c/users/205344a849bc42d19565c4aef9647b97/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:29.690973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.693685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.694058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.740920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] MFA Rules not processed for user `205344a849bc42d19565c4aef9647b97`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:29.742232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:29.761831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.762670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.762978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.763272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.818270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:29.826053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 938/1881] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:29 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:29.839614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.841067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:29.843192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.845259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.854618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff18ffd-3a48-4d68-8ea3-53b9a2df67dd None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:29.858339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:29.858742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 944/1882] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:29.859238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:29.859615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:29.860054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:29.887425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:29.888468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:29.890120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:29.911838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 939/1883] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:29 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:30.032091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa390c6c-3159-4da3-8804-4d8e528f879f service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:30.044438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:30.045168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:30.045479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:30.045803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] PATH_INFO: `/v3/users/44296dff12d34fcfa8b6a42a598a6e05` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:30.048206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44296dff12d34fcfa8b6a42a598a6e05)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:30.051780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:30.097990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:30.109390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560f02ef-f0e6-4425-9e93-b236a11170db service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:30.117624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:30.118387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:30.118668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:30.119793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:30.151625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44296dff12d34fcfa8b6a42a598a6e05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:30.183469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44296dff12d34fcfa8b6a42a598a6e05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:30.203529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44296dff12d34fcfa8b6a42a598a6e05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:30.222606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44296dff12d34fcfa8b6a42a598a6e05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:30.223360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6aeed64e-1593-46e4-be50-f778b038e631 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44296dff12d34fcfa8b6a42a598a6e05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:30.236600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 945/1884] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:29 2022] DELETE /identity/v3/users/44296dff12d34fcfa8b6a42a598a6e05 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:30.242846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:30.247616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:30.247928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:30.281069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:30.282441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:30.291630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:30.293461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:30.308984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 940/1885] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:30.465568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 946/1886] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:30.489544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:30.491306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:30.680222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370f17c5-41b0-4eed-ac58-04728cd9c03b service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:30.709770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:30.800701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7aee35-c850-4a8f-8fa5-dd6dbb7d89f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:30.801430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7aee35-c850-4a8f-8fa5-dd6dbb7d89f9 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:30.803271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7aee35-c850-4a8f-8fa5-dd6dbb7d89f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:30.803566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7aee35-c850-4a8f-8fa5-dd6dbb7d89f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:30.829291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7aee35-c850-4a8f-8fa5-dd6dbb7d89f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:30.832515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7aee35-c850-4a8f-8fa5-dd6dbb7d89f9 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:30.865268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 941/1887] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:31.254818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.256342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:31.266198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:31.273842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f0484-0ea6-442a-b44d-ee6b4c6150d7 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:31.281792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:31.282435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.282712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.283016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] PATH_INFO: `/v3/projects/5a2f835012a944878682c8eecd04d0be` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.284391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a2f835012a944878682c8eecd04d0be)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:31.286256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:31.340631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a2f835012a944878682c8eecd04d0be', 'request_id': 'req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:31.349496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a2f835012a944878682c8eecd04d0be', 'request_id': 'req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:31.355563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a2f835012a944878682c8eecd04d0be', 'request_id': 'req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:31.367403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] The token cache is being invalidate because project 5a2f835012a944878682c8eecd04d0be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:31.367630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:31.368272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-bc5046ad-7e7e-4a7c-9df5-2dc7ea1a8daa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:31.370010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 947/1888] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:31 2022] DELETE /identity/v3/projects/5a2f835012a944878682c8eecd04d0be => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:31.374808 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df6a6bf-64a7-4a33-b8db-01eff03565c6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.375184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df6a6bf-64a7-4a33-b8db-01eff03565c6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.375521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df6a6bf-64a7-4a33-b8db-01eff03565c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.396994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-0df6a6bf-64a7-4a33-b8db-01eff03565c6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:31.398093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0df6a6bf-64a7-4a33-b8db-01eff03565c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:31.468151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 942/1889] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:31.476148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19929672-d456-4553-95e5-347e313910ab None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.476440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19929672-d456-4553-95e5-347e313910ab None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.476772 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19929672-d456-4553-95e5-347e313910ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.497371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-19929672-d456-4553-95e5-347e313910ab None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:31.498676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-19929672-d456-4553-95e5-347e313910ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:31.509962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 948/1890] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:31.511431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df6a6bf-64a7-4a33-b8db-01eff03565c6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.522057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df6a6bf-64a7-4a33-b8db-01eff03565c6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:31.527012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:31.527921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.527921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.528207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.528874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:31.529176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:31.530876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:31.551781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 943/1891] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:31.551884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19929672-d456-4553-95e5-347e313910ab None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.558468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19929672-d456-4553-95e5-347e313910ab None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:31.563389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:31.564123 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.564468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.564806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.565620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:31.567673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:31.597866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 949/1892] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:31 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:31.602047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.612674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084e340f-a379-430b-bc03-079282edc38f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:31.674007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:31.674636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.674919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.675222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.675898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:31.677820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:31.707470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 944/1893] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:31 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:31.708849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.722035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b6d484-dba9-4fc7-a327-f167d94380d5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:31.736013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:31.736715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.737004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.737282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.737993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:31.739706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:31.751431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 950/1894] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:31 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:31.755811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.766779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908588ac-38c1-4578-aec2-662dbfe1b9ef admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:31.771499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f80fd01-1359-4730-815f-acbeffd9e668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:31.772157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80fd01-1359-4730-815f-acbeffd9e668 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.772437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80fd01-1359-4730-815f-acbeffd9e668 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.772771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f80fd01-1359-4730-815f-acbeffd9e668 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.773483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f80fd01-1359-4730-815f-acbeffd9e668 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:31.775106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f80fd01-1359-4730-815f-acbeffd9e668 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:31.789504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 945/1895] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:31 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:31.793716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.806387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb598f-7fce-46f3-b820-39c92d76dcd3 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:31.814728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:31.815968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.815968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.815968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] PATH_INFO: `/v3/projects/c7839c0b004d420893ab5d2819da10be/users/6cdfe913dc604e619c0cc55e7dea1e8e/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.831618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c7839c0b004d420893ab5d2819da10be, user_id=6cdfe913dc604e619c0cc55e7dea1e8e, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:31.834259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:31.849146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 951/1896] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:31 2022] PUT /identity/v3/projects/c7839c0b004d420893ab5d2819da10be/users/6cdfe913dc604e619c0cc55e7dea1e8e/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:31.860130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:31.860570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:31.860917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:31.888550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] MFA Rules not processed for user `6cdfe913dc604e619c0cc55e7dea1e8e`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:31.889804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:31.897988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:31.966992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 946/1897] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:31.975040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:31.976802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:32.040894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af91b038-b6ba-4b64-8f4c-48700a2b68b9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:32.050281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72dbf60-ca8e-420c-8dac-d4dfe45b07c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:32.051147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72dbf60-ca8e-420c-8dac-d4dfe45b07c8 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:32.051894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72dbf60-ca8e-420c-8dac-d4dfe45b07c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:32.052170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72dbf60-ca8e-420c-8dac-d4dfe45b07c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:32.061793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72dbf60-ca8e-420c-8dac-d4dfe45b07c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:32.063397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72dbf60-ca8e-420c-8dac-d4dfe45b07c8 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:32.074193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 952/1898] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:31 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:33.492605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:33.494230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:33.584762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2937836e-334f-42c6-827d-87ebad7aa679 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:33.591947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:33.592630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:33.592896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:33.593173 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] PATH_INFO: `/v3/users/8dbce29803de469cbbcc8a585fea9528` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:33.594351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8dbce29803de469cbbcc8a585fea9528)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:33.596964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:33.652769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8dbce29803de469cbbcc8a585fea9528'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:33.660710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8dbce29803de469cbbcc8a585fea9528'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:33.671605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8dbce29803de469cbbcc8a585fea9528'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:33.686254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8dbce29803de469cbbcc8a585fea9528'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:33.686991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8dbce29803de469cbbcc8a585fea9528'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:33.693943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 947/1899] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:33 2022] DELETE /identity/v3/users/8dbce29803de469cbbcc8a585fea9528 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:33.700273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:33.700842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:33.700842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:33.720712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:33.721920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:33.748565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:33.822781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 953/1900] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:33.833113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:33.834548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:33.868576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:33.897643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b349c50-94d4-40cc-bee4-c8994aa4ac72 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:33.901378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:33.902013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:33.902298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:33.902569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:33.912002 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:33.913456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:33.923360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 948/1901] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:34.329442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:34.331099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:34.344934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ecd4cb-1daa-40bf-b99b-8dfed17883e7 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:34.350853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:34.351749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:34.352256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:34.352776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] PATH_INFO: `/v3/projects/62dde54a654a4239a3f568da765afdad` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:34.354762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62dde54a654a4239a3f568da765afdad)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:34.357174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:34.442613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62dde54a654a4239a3f568da765afdad', 'request_id': 'req-c3657c42-9eb4-40d8-8a22-85ef626ea062'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.453424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62dde54a654a4239a3f568da765afdad', 'request_id': 'req-c3657c42-9eb4-40d8-8a22-85ef626ea062'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.460618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62dde54a654a4239a3f568da765afdad', 'request_id': 'req-c3657c42-9eb4-40d8-8a22-85ef626ea062'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.471353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:34.472820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:34.473803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] The token cache is being invalidate because project 62dde54a654a4239a3f568da765afdad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:34.474267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.475120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.477109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 954/1902] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:34 2022] DELETE /identity/v3/projects/62dde54a654a4239a3f568da765afdad => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:34.481293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:34.484043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:34.486540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ca3e9a-ae40-4d89-af6d-b4dae914a27b service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:34.546663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:34.547363 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:34.547810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:34.548232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] PATH_INFO: `/v3/users/273dff55312c42b7a94316e4d83af9cd` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:34.550899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] RBAC: Authorizing `identity:delete_user(user_id=273dff55312c42b7a94316e4d83af9cd)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:34.553127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:34.555488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3657c42-9eb4-40d8-8a22-85ef626ea062 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:34.561043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:34.561826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:34.562197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:34.562572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] PATH_INFO: `/v3/users/205344a849bc42d19565c4aef9647b97` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:34.564001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] RBAC: Authorizing `identity:delete_user(user_id=205344a849bc42d19565c4aef9647b97)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:34.566082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:34.614130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '273dff55312c42b7a94316e4d83af9cd'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.622090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '273dff55312c42b7a94316e4d83af9cd'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.634710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '273dff55312c42b7a94316e4d83af9cd'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.651587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '205344a849bc42d19565c4aef9647b97'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.653785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:34.653956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '273dff55312c42b7a94316e4d83af9cd'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.654617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87679de5-350a-486b-99a7-6df6d3641847 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '273dff55312c42b7a94316e4d83af9cd'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.663900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 949/1903] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:34 2022] DELETE /identity/v3/users/273dff55312c42b7a94316e4d83af9cd => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:34.668928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:34.669473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:34.670193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:34.672590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '205344a849bc42d19565c4aef9647b97'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.683391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '205344a849bc42d19565c4aef9647b97'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.696210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:34.696494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '205344a849bc42d19565c4aef9647b97'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.697112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '205344a849bc42d19565c4aef9647b97'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:34.698159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:34.710832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 955/1904] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:34 2022] DELETE /identity/v3/users/205344a849bc42d19565c4aef9647b97 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:34.759143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:34.780782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 950/1905] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:34.795104 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:34.797501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:34.899530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce5f447-6fef-45fc-be25-5d97fb1a2916 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:34.906081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3ce577-add6-4287-946a-7f5292c942de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:34.907236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3ce577-add6-4287-946a-7f5292c942de service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:34.907822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3ce577-add6-4287-946a-7f5292c942de service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:34.908111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3ce577-add6-4287-946a-7f5292c942de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:34.985036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3ce577-add6-4287-946a-7f5292c942de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:34.986773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3ce577-add6-4287-946a-7f5292c942de service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.005361 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 956/1906] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:35.050778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.052508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:35.065062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee11506a-cce9-4710-bef1-93ffa1040d0b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.069979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.070671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.073769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.073967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] PATH_INFO: `/v3/projects/367100e46b2547938289a8006735bc4c` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.075283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=367100e46b2547938289a8006735bc4c)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.077371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.181517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '367100e46b2547938289a8006735bc4c', 'request_id': 'req-324e062d-7dae-4108-b048-39adbc0f28d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.192669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '367100e46b2547938289a8006735bc4c', 'request_id': 'req-324e062d-7dae-4108-b048-39adbc0f28d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.200657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '367100e46b2547938289a8006735bc4c', 'request_id': 'req-324e062d-7dae-4108-b048-39adbc0f28d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.214705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] The token cache is being invalidate because project 367100e46b2547938289a8006735bc4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:35.215029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.215692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-324e062d-7dae-4108-b048-39adbc0f28d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.217618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 951/1907] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:35 2022] DELETE /identity/v3/projects/367100e46b2547938289a8006735bc4c => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:35.222545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70e3e9-859d-41e2-92db-875632c4acab None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.222833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70e3e9-859d-41e2-92db-875632c4acab None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.223136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70e3e9-859d-41e2-92db-875632c4acab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.245018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4c70e3e9-859d-41e2-92db-875632c4acab None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:35.246180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4c70e3e9-859d-41e2-92db-875632c4acab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:35.335253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 957/1908] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:35.337127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.337594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.338013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.374396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:35.376332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:35.391283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 952/1909] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:35.400232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70e3e9-859d-41e2-92db-875632c4acab None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.415612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70e3e9-859d-41e2-92db-875632c4acab None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.422365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.423094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.423495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.423682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.428376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.430048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:35.433758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:35.434138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.444969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.458105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea4015f-1918-4d42-82c8-7a91bfeaa854 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.471783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.477392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 958/1910] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:35.478432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.478868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.479824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] PATH_INFO: `/v3/projects/cdee78f3a79e4e7dab0cedd5aa21cac6` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.482013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.483065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] RBAC: Authorizing `identity:delete_project(project_id=cdee78f3a79e4e7dab0cedd5aa21cac6)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.487483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.496562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:35.509497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db228a0-5a5d-486a-9f83-4fa0b0b381a9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.548728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.549456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.549741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.550056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.550828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.553173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.575500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cdee78f3a79e4e7dab0cedd5aa21cac6', 'request_id': 'req-8ed61d89-ce1c-4854-86b8-c5535dba980b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.589047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cdee78f3a79e4e7dab0cedd5aa21cac6', 'request_id': 'req-8ed61d89-ce1c-4854-86b8-c5535dba980b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.589269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 959/1911] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:35 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:35.590243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.592845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cdee78f3a79e4e7dab0cedd5aa21cac6', 'request_id': 'req-8ed61d89-ce1c-4854-86b8-c5535dba980b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.605537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] The token cache is being invalidate because project cdee78f3a79e4e7dab0cedd5aa21cac6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:35.605838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.606523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.608508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 953/1912] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:35 2022] DELETE /identity/v3/projects/cdee78f3a79e4e7dab0cedd5aa21cac6 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:35.612321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.613845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:35.658194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73ff745-2c78-4445-999e-40a2d879973b admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.663164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.663797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.664090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.664356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.665053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.666993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.671854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:35.688697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed61d89-ce1c-4854-86b8-c5535dba980b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.695790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.696538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.696883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.697218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] PATH_INFO: `/v3/users/d65864b791a344a3af132ad56a276bfa` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.697695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 960/1913] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:35 2022] POST /identity/v3/users => generated 293 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:35.700512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=d65864b791a344a3af132ad56a276bfa)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.702657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.704236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.707550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:35.720920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7a266-e820-410a-97fa-a542a03154fc admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.726043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.726686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.727015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.727306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.728028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.729730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.748615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 961/1914] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:35 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:35.751490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.766618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd65864b791a344a3af132ad56a276bfa'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.767735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd465b0-7b0b-448d-a07b-09207646f777 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.773236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.773931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.774261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.774540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.774727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd65864b791a344a3af132ad56a276bfa'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.775595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.778706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.791834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd65864b791a344a3af132ad56a276bfa'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.796817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 962/1915] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:35 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:35.800676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:35.809375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd65864b791a344a3af132ad56a276bfa'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.810202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-196e83a6-13cb-45d8-ad04-d1366fcb71cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd65864b791a344a3af132ad56a276bfa'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:35.826927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae3536d-3d57-4661-afdf-f678b57182bf admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:35.832273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 954/1916] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:35 2022] DELETE /identity/v3/users/d65864b791a344a3af132ad56a276bfa => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:35.921943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:35.922645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.923464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.923464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] PATH_INFO: `/v3/projects/86cf1121794c454b8690c0d7085cc2c2/users/5923f7e8a8b54cf1848b3297950aa17e/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:35.946901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=86cf1121794c454b8690c0d7085cc2c2, user_id=5923f7e8a8b54cf1848b3297950aa17e, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:35.949696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:35.969946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 963/1917] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:35 2022] PUT /identity/v3/projects/86cf1121794c454b8690c0d7085cc2c2/users/5923f7e8a8b54cf1848b3297950aa17e/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:35.986835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:35.988599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:35.990572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.057177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] MFA Rules not processed for user `5923f7e8a8b54cf1848b3297950aa17e`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:36.058400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.142933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 955/1918] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:36.147592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.149815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.189179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:36.232611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1e280-81b2-4575-ab23-e3bded62bbbb admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.237322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.238072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.238353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.238629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.239414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.241842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.258404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.259934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.287896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52ddab0-4280-4f7c-9361-604eb12db21e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.294036 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.295171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.300066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 964/1919] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:36 2022] POST /identity/v3/projects => generated 387 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:36.301273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.302051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.302368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] PATH_INFO: `/v3/projects/d3fe38e8d7fa4639aa60738319c61177` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.303715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3fe38e8d7fa4639aa60738319c61177)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.307777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.383365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3fe38e8d7fa4639aa60738319c61177', 'request_id': 'req-240ba86d-0142-46e5-a5bb-5fa49109be85'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:36.387934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:36.393581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224be77c-39d0-4834-bd13-12b60a7192a0 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.394546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3fe38e8d7fa4639aa60738319c61177', 'request_id': 'req-240ba86d-0142-46e5-a5bb-5fa49109be85'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:36.402857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3fe38e8d7fa4639aa60738319c61177', 'request_id': 'req-240ba86d-0142-46e5-a5bb-5fa49109be85'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:36.432263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] The token cache is being invalidate because project d3fe38e8d7fa4639aa60738319c61177 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:36.432556 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:36.433231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-240ba86d-0142-46e5-a5bb-5fa49109be85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:36.435254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 956/1920] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:36 2022] DELETE /identity/v3/projects/d3fe38e8d7fa4639aa60738319c61177 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:36.441372 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909980e4-8b64-4903-8be5-b6bf2cda03b3 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.442100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909980e4-8b64-4903-8be5-b6bf2cda03b3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.443111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909980e4-8b64-4903-8be5-b6bf2cda03b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.466969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-909980e4-8b64-4903-8be5-b6bf2cda03b3 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:36.468212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-909980e4-8b64-4903-8be5-b6bf2cda03b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.473562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.474205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.474490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.475195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.475876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.477930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.502658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 965/1921] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:36 2022] POST /identity/v3/users => generated 300 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:36.503701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.505275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.517059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533b387a-6dc2-4fae-a65d-9d36ded18da5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.521503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.522103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.522387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.522644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.523368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.525003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.532280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 957/1922] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:36.538990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 966/1923] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:36.540858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.541839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5dcaab-b3ba-464f-b255-a49d27f827e3 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.542174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5dcaab-b3ba-464f-b255-a49d27f827e3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.542530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5dcaab-b3ba-464f-b255-a49d27f827e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.551625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c562f0-c8ee-488f-971b-291ec94b2f3e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.556870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.558050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.558050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.558236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] PATH_INFO: `/v3/projects/a69bfbb0f0804d73939a2a27a4769919/users/1be1b68411924223a02b39f3132595c2/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.571119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9f5dcaab-b3ba-464f-b255-a49d27f827e3 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:36.572352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9f5dcaab-b3ba-464f-b255-a49d27f827e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.577885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a69bfbb0f0804d73939a2a27a4769919, user_id=1be1b68411924223a02b39f3132595c2, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.581643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.583796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 958/1924] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:36.587920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5dcaab-b3ba-464f-b255-a49d27f827e3 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.599194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 967/1925] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:36 2022] PUT /identity/v3/projects/a69bfbb0f0804d73939a2a27a4769919/users/1be1b68411924223a02b39f3132595c2/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:36.602615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5dcaab-b3ba-464f-b255-a49d27f827e3 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.603761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.637533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:36.663520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.664158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.664438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.664826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.665539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:36.665847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.667502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.680593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 959/1926] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:36.684768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.696411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eece9f47-c07e-43d0-bd4f-8b65ec37bbfa admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.696911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1795b397-fe13-4202-96cf-9296089b2c9e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.701406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.702357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.702655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.703301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.703998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.705996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.708557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.709407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.709537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.710841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.714912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.717351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.737988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 968/1927] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:36.746110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.746814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 960/1928] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:36 2022] POST /identity/v3/projects => generated 369 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:36.753059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.757281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6c1996-7758-4cef-bdd6-6df2a2476024 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.812888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:36.817140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ecea38-7203-4405-b26e-8b5ae850bda5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.817871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ecea38-7203-4405-b26e-8b5ae850bda5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.818249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ecea38-7203-4405-b26e-8b5ae850bda5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.818516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ecea38-7203-4405-b26e-8b5ae850bda5 admin admin] PATH_INFO: `/v3/projects/a69bfbb0f0804d73939a2a27a4769919/users/1be1b68411924223a02b39f3132595c2/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.820641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ecea38-7203-4405-b26e-8b5ae850bda5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a69bfbb0f0804d73939a2a27a4769919, user_id=1be1b68411924223a02b39f3132595c2, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.823422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ecea38-7203-4405-b26e-8b5ae850bda5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.828758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decb5894-6ffa-46d7-b6f7-cc4ed987c0ae admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.834392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 969/1929] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:36 2022] PUT /identity/v3/projects/a69bfbb0f0804d73939a2a27a4769919/users/1be1b68411924223a02b39f3132595c2/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:36.845092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.845426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.845754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.874855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] MFA Rules not processed for user `1be1b68411924223a02b39f3132595c2`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:36.876129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.887605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.888239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.888869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.888869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.889499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.891491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.913872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 961/1930] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:36 2022] POST /identity/v3/users => generated 285 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:36.918043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.928464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eadd08-2cd5-4f9b-9e7a-993ac1e17e14 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.933296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.933919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.934196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.934472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.935195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.936938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.947935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 962/1931] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:36.951877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.961528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 970/1932] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:36 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:36.962533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bf7992-ebb3-411b-b165-9d8e55225e90 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:36.969977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:36.970820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:36.971593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:36.972185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:36.973041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:36.973622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:36.975121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:36.975268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:36.989349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 963/1933] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:36.990702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:37.005961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2becb8d3-4d3f-4dc2-a102-7c1ed39f43f1 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:37.010995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e9550-3c2c-449b-8e5e-6b0973e290ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:37.011584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e9550-3c2c-449b-8e5e-6b0973e290ed admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.011714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e9550-3c2c-449b-8e5e-6b0973e290ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.011953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e9550-3c2c-449b-8e5e-6b0973e290ed admin admin] PATH_INFO: `/v3/projects/0bc30038bf074e49a8f72bb78e4faa85/users/e653144bafa047b7931942fa5961422c/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.029604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e9550-3c2c-449b-8e5e-6b0973e290ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=0bc30038bf074e49a8f72bb78e4faa85, user_id=e653144bafa047b7931942fa5961422c, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:37.030476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:37.033299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e9550-3c2c-449b-8e5e-6b0973e290ed admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:37.051022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 964/1934] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:36 2022] PUT /identity/v3/projects/0bc30038bf074e49a8f72bb78e4faa85/users/e653144bafa047b7931942fa5961422c/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:37.053474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1ae83c-a916-422e-a128-ec7d5526d6c6 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:37.061077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.061370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.063487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.122098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:37.122919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.123433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.123565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] MFA Rules not processed for user `e653144bafa047b7931942fa5961422c`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:37.123764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.125458 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:37.229558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 965/1935] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:37 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:37.256739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:37.258579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:37.271900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3500556-ec9a-4e4f-ae42-63d8245e73df None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:37.274235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:37.275574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:37.276203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.276481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.276753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.280350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:37.289934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:37.292930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:37.296943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 971/1936] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:36 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:37.326971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 966/1937] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:37 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:37.340018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:37.428270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2216b2d8-1740-4db3-9b4c-7d79c541a8c5 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:37.434986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:37.435749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.436026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.436369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] PATH_INFO: `/v3/users/e653144bafa047b7931942fa5961422c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.439921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=e653144bafa047b7931942fa5961422c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:37.441843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:37.455612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:37.472699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477b4233-2dab-4600-8bcd-f9a2977cc3a1 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:37.481682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:37.483794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.484328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.484833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] PATH_INFO: `/v3/users/5923f7e8a8b54cf1848b3297950aa17e` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.487905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5923f7e8a8b54cf1848b3297950aa17e)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:37.493141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:37.505359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:37.509084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:37.511632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e653144bafa047b7931942fa5961422c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.531970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e653144bafa047b7931942fa5961422c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.543529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e653144bafa047b7931942fa5961422c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.556188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e653144bafa047b7931942fa5961422c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.557821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e6d5f25c-c1fb-4152-a14d-065197323fbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e653144bafa047b7931942fa5961422c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.569047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5923f7e8a8b54cf1848b3297950aa17e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.573610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 972/1938] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:37 2022] DELETE /identity/v3/users/e653144bafa047b7931942fa5961422c => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:37.578706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.579042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.579568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.581228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5923f7e8a8b54cf1848b3297950aa17e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.594189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5923f7e8a8b54cf1848b3297950aa17e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.602271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:37.603681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:37.611842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5923f7e8a8b54cf1848b3297950aa17e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.612565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-81545c07-8158-438b-923b-6b010d0fc95d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5923f7e8a8b54cf1848b3297950aa17e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:37.620366 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 967/1939] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:37 2022] DELETE /identity/v3/users/5923f7e8a8b54cf1848b3297950aa17e => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:37.625566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.625566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.625857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.650982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:37.652206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:37.696078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 973/1940] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:37.709194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:37.710824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:37.734331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 968/1941] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:37.760450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:37.761976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:37.808345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e157d4af-9c15-41a0-9b62-2a9eb45b1092 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:37.911379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c64e9-1a09-4898-a5ff-eeaed79a0e0b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:37.915504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:37.916151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.916439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.916709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.934345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:37.936446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:37.940499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:37.942094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:37.943212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:37.943828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:37.969072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:37.971078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:37.972550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 969/1942] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:38.003181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 974/1943] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:38.538912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:38.540400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:38.555299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82f00bb-6699-4e09-ac62-aefe35e7cc4e service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:38.575199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:38.576557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:38.576845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:38.577251 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] PATH_INFO: `/v3/projects/0bc30038bf074e49a8f72bb78e4faa85` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:38.579644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0bc30038bf074e49a8f72bb78e4faa85)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:38.583232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:38.606401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:38.608021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:38.633944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2de0b85-6e64-476e-a562-fd19ac4435bc service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:38.641471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:38.655323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:38.657461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:38.657770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:38.659914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] PATH_INFO: `/v3/projects/86cf1121794c454b8690c0d7085cc2c2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:38.663155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=86cf1121794c454b8690c0d7085cc2c2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:38.665179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:38.688216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:38.705962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0bc30038bf074e49a8f72bb78e4faa85', 'request_id': 'req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.727959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0bc30038bf074e49a8f72bb78e4faa85', 'request_id': 'req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.736198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0bc30038bf074e49a8f72bb78e4faa85', 'request_id': 'req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.755954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86cf1121794c454b8690c0d7085cc2c2', 'request_id': 'req-9e96ed26-ee56-476d-b34b-3ef293d3ded5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.758012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] The token cache is being invalidate because project 0bc30038bf074e49a8f72bb78e4faa85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:38.758505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.759404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c1891c08-603b-4f5d-b3e1-3b5aa4ec67b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.762203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 970/1944] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:38 2022] DELETE /identity/v3/projects/0bc30038bf074e49a8f72bb78e4faa85 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:38.771645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86cf1121794c454b8690c0d7085cc2c2', 'request_id': 'req-9e96ed26-ee56-476d-b34b-3ef293d3ded5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.773446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae907588-33e3-4d3d-9ec5-169fa8425023 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:38.773790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae907588-33e3-4d3d-9ec5-169fa8425023 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:38.774125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae907588-33e3-4d3d-9ec5-169fa8425023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:38.787405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86cf1121794c454b8690c0d7085cc2c2', 'request_id': 'req-9e96ed26-ee56-476d-b34b-3ef293d3ded5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.803906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] The token cache is being invalidate because project 86cf1121794c454b8690c0d7085cc2c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:38.804204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.805234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.810940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 975/1945] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:38 2022] DELETE /identity/v3/projects/86cf1121794c454b8690c0d7085cc2c2 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:38.813151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:38.814755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:38.820717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ae907588-33e3-4d3d-9ec5-169fa8425023 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:38.823721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ae907588-33e3-4d3d-9ec5-169fa8425023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:38.892291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e96ed26-ee56-476d-b34b-3ef293d3ded5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:38.897693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:38.899094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:38.899515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:38.899799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] PATH_INFO: `/v3/users/1be1b68411924223a02b39f3132595c2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:38.901000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1be1b68411924223a02b39f3132595c2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:38.902994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 971/1946] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:38.903379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:38.909737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaef039e-f7a9-423e-97dd-2d3d413f3228 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:38.910153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaef039e-f7a9-423e-97dd-2d3d413f3228 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:38.910523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaef039e-f7a9-423e-97dd-2d3d413f3228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:38.933507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-aaef039e-f7a9-423e-97dd-2d3d413f3228 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:38.935085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-aaef039e-f7a9-423e-97dd-2d3d413f3228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:38.943982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 972/1947] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:38.949358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaef039e-f7a9-423e-97dd-2d3d413f3228 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:38.960975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaef039e-f7a9-423e-97dd-2d3d413f3228 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:38.965497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1be1b68411924223a02b39f3132595c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.970856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:38.972257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:38.972555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:38.972903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:38.973693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:38.974629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:38.977271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:38.983406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1be1b68411924223a02b39f3132595c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:38.999389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1be1b68411924223a02b39f3132595c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.001941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 973/1948] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:39.003253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:39.015769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e220aa85-e2a6-4308-a83d-f70e2d3f0576 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:39.019650 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1be1b68411924223a02b39f3132595c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.020275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1be1b68411924223a02b39f3132595c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.022807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:39.023591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.023941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.024223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.024902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:39.027061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:39.027498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 976/1949] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:38 2022] DELETE /identity/v3/users/1be1b68411924223a02b39f3132595c2 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:39.067303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 974/1950] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:39 2022] POST /identity/v3/projects => generated 365 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:39.071774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:39.073345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:39.102632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:39.150131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479f22d-c72a-46db-a501-8e194886523f admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:39.233863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:39.234565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.234855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.235158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.235841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:39.237746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:39.267934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 977/1951] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:39 2022] POST /identity/v3/users => generated 283 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:39.272586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:39.285382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d8a9af-b7f8-499b-bf27-54f39024f48f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:39.290389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:39.291125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.291414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.291836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.292570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:39.294402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:39.307473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 975/1952] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:39.311335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:39.327429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d517c9b8-3cc9-4fec-a100-0161745ae20d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:39.333717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029fe6e8-372c-432d-b6df-64d85c7fe1e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:39.334582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029fe6e8-372c-432d-b6df-64d85c7fe1e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.334870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029fe6e8-372c-432d-b6df-64d85c7fe1e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.335753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029fe6e8-372c-432d-b6df-64d85c7fe1e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.336529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029fe6e8-372c-432d-b6df-64d85c7fe1e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:39.338254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029fe6e8-372c-432d-b6df-64d85c7fe1e6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:39.352214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 978/1953] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:39 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:39.356239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:39.370405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a1e22b-a9ed-4deb-aeeb-a52a15830a01 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:39.376041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:39.377253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.377716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.378163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] PATH_INFO: `/v3/projects/8327b64aa27b439bbacb285d1b4edbde/users/d2e5f6a317494f14808b1036d50e1b0f/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.402044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8327b64aa27b439bbacb285d1b4edbde, user_id=d2e5f6a317494f14808b1036d50e1b0f, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:39.405103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:39.423508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 976/1954] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:39 2022] PUT /identity/v3/projects/8327b64aa27b439bbacb285d1b4edbde/users/d2e5f6a317494f14808b1036d50e1b0f/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:39.434760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.435122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.435402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.477286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] MFA Rules not processed for user `d2e5f6a317494f14808b1036d50e1b0f`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:39.478751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:39.560176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 979/1955] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:39 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:39.572813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:39.574836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:39.617462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:39.619212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:39.644794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:39.668167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9081e9f5-c0b0-4780-b6c3-24815dbfda39 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:39.678349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20792cf9-e211-48ba-8f36-071c4568d4ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:39.680631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:39.681325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20792cf9-e211-48ba-8f36-071c4568d4ec service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.681818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20792cf9-e211-48ba-8f36-071c4568d4ec service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.682280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20792cf9-e211-48ba-8f36-071c4568d4ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.707850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20792cf9-e211-48ba-8f36-071c4568d4ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:39.709515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20792cf9-e211-48ba-8f36-071c4568d4ec service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:39.725777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84a172-83b9-41a4-b690-fbc3e1f1528c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:39.728411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 977/1956] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:39 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:39.730750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:39.731448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.731735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.732000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] PATH_INFO: `/v3/projects/a69bfbb0f0804d73939a2a27a4769919` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.733927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a69bfbb0f0804d73939a2a27a4769919)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:39.736161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:39.801406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a69bfbb0f0804d73939a2a27a4769919', 'request_id': 'req-ab646014-93b0-45cc-a8de-1e62c35b7296'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.814114 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a69bfbb0f0804d73939a2a27a4769919', 'request_id': 'req-ab646014-93b0-45cc-a8de-1e62c35b7296'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.827783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a69bfbb0f0804d73939a2a27a4769919', 'request_id': 'req-ab646014-93b0-45cc-a8de-1e62c35b7296'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.835201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] The token cache is being invalidate because project a69bfbb0f0804d73939a2a27a4769919 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:39.835489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.836148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-ab646014-93b0-45cc-a8de-1e62c35b7296 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:39.837983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 980/1957] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:39 2022] DELETE /identity/v3/projects/a69bfbb0f0804d73939a2a27a4769919 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:39.843594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c286c96-7dc7-4cb4-b9b1-d3c9e25d3dcc None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:39.844237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c286c96-7dc7-4cb4-b9b1-d3c9e25d3dcc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:39.845167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c286c96-7dc7-4cb4-b9b1-d3c9e25d3dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:39.883958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2c286c96-7dc7-4cb4-b9b1-d3c9e25d3dcc None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:39.886114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2c286c96-7dc7-4cb4-b9b1-d3c9e25d3dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:39.999266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 978/1958] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:40.006761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b2b7ef-1e18-4021-9ab7-cf1eb48cfe67 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.007186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b2b7ef-1e18-4021-9ab7-cf1eb48cfe67 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.007546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b2b7ef-1e18-4021-9ab7-cf1eb48cfe67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.036638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-32b2b7ef-1e18-4021-9ab7-cf1eb48cfe67 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:40.038387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-32b2b7ef-1e18-4021-9ab7-cf1eb48cfe67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:40.052800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 981/1959] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:40.056588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c286c96-7dc7-4cb4-b9b1-d3c9e25d3dcc None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.070908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c286c96-7dc7-4cb4-b9b1-d3c9e25d3dcc None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.079183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.080422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.080956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.081435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.082646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:40.083200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.085393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.113925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 979/1960] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:40.114724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b2b7ef-1e18-4021-9ab7-cf1eb48cfe67 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.125039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b2b7ef-1e18-4021-9ab7-cf1eb48cfe67 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.132276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.133307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.133590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.133906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.134586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.137894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.179055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.179323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 982/1961] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/projects => generated 375 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:40.192485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f250e97-73dc-47d5-9638-300cfdfc9b70 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.233549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.235428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:40.258021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.258860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.259351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.259775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.260771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.265735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.310735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 980/1962] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/users => generated 288 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:40.311749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.337750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514f897-72b2-457b-9e5b-9baedc79bd6a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.345750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.346485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.347184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.347467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.348325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.351724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.367702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 981/1963] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:40 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:40.369931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43c025e-3e05-4b6b-914b-8a570e8e7fa6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.371857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.375312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.376090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.376548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.376890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] PATH_INFO: `/v3/users/6cdfe913dc604e619c0cc55e7dea1e8e` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.378105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cdfe913dc604e619c0cc55e7dea1e8e)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.380202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.390052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c343480d-f0ed-444a-9ad7-6413b5f3ce52 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.397018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.397982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.398846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.399318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.400331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.402310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.417785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 982/1964] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:40 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:40.422267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.432483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d975e17-252c-4275-9cad-1234e003770d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.437906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfbfc9e-9260-479a-8607-219a4e00ca06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.438580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbfc9e-9260-479a-8607-219a4e00ca06 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.438714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:40.438883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbfc9e-9260-479a-8607-219a4e00ca06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.439173 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbfc9e-9260-479a-8607-219a4e00ca06 admin admin] PATH_INFO: `/v3/projects/0b98732cfa174357a82beef1a2af41c2/users/7dc0bd3be2dc4c4898aa94589a2037d5/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.446474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cdfe913dc604e619c0cc55e7dea1e8e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:40.455423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfbfc9e-9260-479a-8607-219a4e00ca06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b98732cfa174357a82beef1a2af41c2, user_id=7dc0bd3be2dc4c4898aa94589a2037d5, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.458094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfbfc9e-9260-479a-8607-219a4e00ca06 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.464928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cdfe913dc604e619c0cc55e7dea1e8e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:40.473258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cdfe913dc604e619c0cc55e7dea1e8e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:40.474969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 983/1965] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:40 2022] PUT /identity/v3/projects/0b98732cfa174357a82beef1a2af41c2/users/7dc0bd3be2dc4c4898aa94589a2037d5/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:40.483710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cdfe913dc604e619c0cc55e7dea1e8e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:40.484880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4a928ea3-2e45-4cf2-b1cc-183bc7008776 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cdfe913dc604e619c0cc55e7dea1e8e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:40.488238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.488238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.488238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.494895 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 983/1966] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:40 2022] DELETE /identity/v3/users/6cdfe913dc604e619c0cc55e7dea1e8e => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:40.499643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.499935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.500294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.513532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:40.521653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:40.523202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:40.524540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] MFA Rules not processed for user `7dc0bd3be2dc4c4898aa94589a2037d5`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:40.526001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:40.596593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 984/1967] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:40.602013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 984/1968] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:40.606610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.608054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:40.608800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.610694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:40.621320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0bc1ba-9c80-4032-a148-37bc52383c38 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.627672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.628478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.628819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.629203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.630029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.632255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.668068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 985/1969] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/projects => generated 379 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:40.672668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.689941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039004e8-41c3-4113-9748-903e0eb56523 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.759044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.759827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.760180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.760967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.765515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52de0d7f-dc1b-4fcf-ac44-c98ac7469527 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.772224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.772356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.775233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.775233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.775233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.775520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.775600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.778368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.787402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 985/1970] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:40.812792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 986/1971] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/users => generated 290 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:40.818194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.833386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f29fb1-8494-4a83-a6c1-9f75fbd0da16 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.854843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.855496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.855790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.856085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.857285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.859005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.883074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 986/1972] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:40 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:40.887568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.900019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ee9d24-2359-4683-acb8-76074061b15a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.907194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d22a081-eb19-47ca-9d7f-71b69b10ea25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.907941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d22a081-eb19-47ca-9d7f-71b69b10ea25 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.908295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d22a081-eb19-47ca-9d7f-71b69b10ea25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.908746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d22a081-eb19-47ca-9d7f-71b69b10ea25 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.909533 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d22a081-eb19-47ca-9d7f-71b69b10ea25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.911369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d22a081-eb19-47ca-9d7f-71b69b10ea25 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.927929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:40.928483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 987/1973] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:40 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:40.941141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a1b1ee-a2e7-4288-b795-87d6a4ecbe59 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:40.946533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:40.947357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.947707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.948049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] PATH_INFO: `/v3/projects/23609d1827aa41db8ff53c079832cf28/users/3eefcb5387354918b82a3ac1e4454838/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:40.966584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] RBAC: Authorizing `identity:create_grant(project_id=23609d1827aa41db8ff53c079832cf28, user_id=3eefcb5387354918b82a3ac1e4454838, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:40.969748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:40.985360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 987/1974] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:40 2022] PUT /identity/v3/projects/23609d1827aa41db8ff53c079832cf28/users/3eefcb5387354918b82a3ac1e4454838/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:40.996221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a646b-cb2e-4848-b942-d91688c88a98 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:40.996513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a646b-cb2e-4848-b942-d91688c88a98 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:40.996796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a646b-cb2e-4848-b942-d91688c88a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.053309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-6a9a646b-cb2e-4848-b942-d91688c88a98 None None] MFA Rules not processed for user `3eefcb5387354918b82a3ac1e4454838`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:41.055467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6a9a646b-cb2e-4848-b942-d91688c88a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:41.060759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:41.143948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 988/1975] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:40 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:41.148749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:41.195568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:41.243536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2208b59b-d293-4852-888d-26d0d7d2b08b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:41.248435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:41.249069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.249349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.249625 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.250305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:41.252240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:41.292675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 988/1976] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:41 2022] POST /identity/v3/projects => generated 379 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:41.296819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a646b-cb2e-4848-b942-d91688c88a98 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:41.310181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a646b-cb2e-4848-b942-d91688c88a98 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:41.409985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:41.410634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.410923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.411307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.412010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:41.414841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:41.451460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 989/1977] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:41 2022] POST /identity/v3/users => generated 296 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:41.456043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:41.469597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111ede26-17ba-4161-8605-78a25474968f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:41.474591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:41.475377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.475806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.476352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.477354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:41.479266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:41.493968 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 989/1978] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:41 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:41.501248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:41.527034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f6d954-d053-48bf-8d70-0596fe34f890 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:41.534705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:41.543447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.543860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.544231 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] PATH_INFO: `/v3/projects/224696e362c34e0e8a6558b2e456d7c4/users/fe999e9fa73b4ba9850a7510e39fb3a1/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.577332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=224696e362c34e0e8a6558b2e456d7c4, user_id=fe999e9fa73b4ba9850a7510e39fb3a1, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:41.582848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:41.595540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:41.597286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:41.617657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aef6a1a-05a0-4549-ae96-113e5a2e9681 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:41.618587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 990/1979] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:41 2022] PUT /identity/v3/projects/224696e362c34e0e8a6558b2e456d7c4/users/fe999e9fa73b4ba9850a7510e39fb3a1/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:41.625167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:41.717812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2628d5b-6c2f-48c9-a4e0-9d0763408a3f admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:41.732127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:41.732127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.732127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.733312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.733876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:41.736657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:41.753129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 991/1980] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:41 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:41.757484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:41.783987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cc6e59-8eb7-48fb-8c2d-8d7a5b9900ed admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:41.794053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6327e758-c467-4791-b6cd-ed091ed497ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:41.795139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327e758-c467-4791-b6cd-ed091ed497ba admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.795602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327e758-c467-4791-b6cd-ed091ed497ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.797753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6327e758-c467-4791-b6cd-ed091ed497ba admin admin] PATH_INFO: `/v3/projects/224696e362c34e0e8a6558b2e456d7c4/users/fe999e9fa73b4ba9850a7510e39fb3a1/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.801622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:41.802299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.802591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.803054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] PATH_INFO: `/v3/projects/c7839c0b004d420893ab5d2819da10be` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.806737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6327e758-c467-4791-b6cd-ed091ed497ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=224696e362c34e0e8a6558b2e456d7c4, user_id=fe999e9fa73b4ba9850a7510e39fb3a1, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:41.806907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c7839c0b004d420893ab5d2819da10be)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:41.808755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:41.809239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6327e758-c467-4791-b6cd-ed091ed497ba admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:41.817382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:41.837672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 992/1981] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:41 2022] PUT /identity/v3/projects/224696e362c34e0e8a6558b2e456d7c4/users/fe999e9fa73b4ba9850a7510e39fb3a1/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:41.849710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:41.850248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:41.850584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:41.940692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] MFA Rules not processed for user `fe999e9fa73b4ba9850a7510e39fb3a1`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:41.944887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:41.957494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c7839c0b004d420893ab5d2819da10be', 'request_id': 'req-93ef640d-c698-4473-b190-685a07fe4496'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:41.987522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c7839c0b004d420893ab5d2819da10be', 'request_id': 'req-93ef640d-c698-4473-b190-685a07fe4496'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:41.999032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c7839c0b004d420893ab5d2819da10be', 'request_id': 'req-93ef640d-c698-4473-b190-685a07fe4496'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:42.029459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] The token cache is being invalidate because project c7839c0b004d420893ab5d2819da10be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:42.029800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:42.030525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-93ef640d-c698-4473-b190-685a07fe4496 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:42.036794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 990/1982] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:41 2022] DELETE /identity/v3/projects/c7839c0b004d420893ab5d2819da10be => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:42.042042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a66f3-a848-4db0-8afe-5198660c7915 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.042042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a66f3-a848-4db0-8afe-5198660c7915 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.042316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a66f3-a848-4db0-8afe-5198660c7915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.086387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-dd6a66f3-a848-4db0-8afe-5198660c7915 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:42.090360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dd6a66f3-a848-4db0-8afe-5198660c7915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:42.145575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 993/1983] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:41 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:42.145778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:42.153837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.155491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:42.222085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 991/1984] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:42.230312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563c1c4-93bc-4084-bcc9-4b796016cab1 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.230689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563c1c4-93bc-4084-bcc9-4b796016cab1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.233616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563c1c4-93bc-4084-bcc9-4b796016cab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.256783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c563c1c4-93bc-4084-bcc9-4b796016cab1 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:42.258146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c563c1c4-93bc-4084-bcc9-4b796016cab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:42.260789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c40a0b5-2fd9-412e-8659-235927901cd3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.266724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.267524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 992/1985] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:42.268346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.268768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.269541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.271408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563c1c4-93bc-4084-bcc9-4b796016cab1 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.281686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563c1c4-93bc-4084-bcc9-4b796016cab1 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.287753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.288657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.289177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.289567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.290183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.290623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:42.291048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.292443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.293447 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.311635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 994/1986] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:42 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:42.326562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 993/1987] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:11:42.332832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.356379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f032b2df-90d2-4f5a-b7a1-d4df9f7fd9f2 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.364006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.366233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.367418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.367418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.367709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.371443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.377018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.379608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:42.394206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3bb503-019c-4ff3-a4fb-b38b7c1dbca2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.399224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.400195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.400858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.401333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.403872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:42.443281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 995/1988] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:42 2022] POST /identity/v3/projects => generated 385 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:42.448096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.487620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.489832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.507220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 994/1989] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:42 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:42.581159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1278ce0-505d-4453-ae19-32a3b5859ac7 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.587651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.588455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.588774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.589057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.590175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.593738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.605312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:42.626534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 996/1990] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:42 2022] POST /identity/v3/users => generated 293 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:42.630756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.652974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaccbfee-4d43-4f53-9112-1e9ea3206ba1 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.661558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.662285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.662593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.662867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.663764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.665913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.679548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 995/1991] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:42 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:42.683287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.695755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b7429a-036b-484b-97a1-7b592a01ee7d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.701662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bb7232-920c-4012-948f-9ea2a2f0479b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.702561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bb7232-920c-4012-948f-9ea2a2f0479b admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.703328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bb7232-920c-4012-948f-9ea2a2f0479b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.703697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bb7232-920c-4012-948f-9ea2a2f0479b admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.704625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bb7232-920c-4012-948f-9ea2a2f0479b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.707112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bb7232-920c-4012-948f-9ea2a2f0479b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.723940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 997/1992] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:42 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:42.729973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:42.745817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b388ed1-78e5-4aa7-afa7-61c87b3e9606 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:42.751039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:42.751695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.752020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.752659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] PATH_INFO: `/v3/projects/9271e6aaf1ea446888d612cf5b40000b/users/9f3d81b4a0d24e3ea7e8dcd841aa4775/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.777062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9271e6aaf1ea446888d612cf5b40000b, user_id=9f3d81b4a0d24e3ea7e8dcd841aa4775, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:42.781880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:42.810326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 996/1993] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:42 2022] PUT /identity/v3/projects/9271e6aaf1ea446888d612cf5b40000b/users/9f3d81b4a0d24e3ea7e8dcd841aa4775/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:42.822108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:42.822439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:42.823304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:42.856100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] MFA Rules not processed for user `9f3d81b4a0d24e3ea7e8dcd841aa4775`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:42.857542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:42.993591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 998/1994] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:42 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:43.003799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:43.094124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445dc284-b541-4df3-8b75-629750d9bcd3 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:43.099909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:43.100934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:43.101218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:43.101505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:43.111704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:43.113214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:43.125048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 997/1995] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:43 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:45.672312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:45.674285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:45.749843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d063cb3-3b3b-4450-8aa2-4e4e56112e3b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:45.755268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:45.755926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:45.756229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:45.756469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] PATH_INFO: `/v3/users/7dc0bd3be2dc4c4898aa94589a2037d5` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:45.757635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] RBAC: Authorizing `identity:delete_user(user_id=7dc0bd3be2dc4c4898aa94589a2037d5)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:45.759694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:45.815695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:45.821459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7dc0bd3be2dc4c4898aa94589a2037d5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.835968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7dc0bd3be2dc4c4898aa94589a2037d5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.841923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9376fdf2-454f-4309-84b7-a5c7e6fc27a8 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:45.844307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=3,keystone:SELECT=88,keystone:DELETE=9 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:45.848827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7dc0bd3be2dc4c4898aa94589a2037d5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.852882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:45.855213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:45.856959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:45.857441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] PATH_INFO: `/v3/users/9f3d81b4a0d24e3ea7e8dcd841aa4775` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:45.858712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f3d81b4a0d24e3ea7e8dcd841aa4775)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:45.861279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:45.863219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7dc0bd3be2dc4c4898aa94589a2037d5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.864263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-449c4925-4f63-4211-b100-43e89908d20e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7dc0bd3be2dc4c4898aa94589a2037d5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.874619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:45.874868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 999/1996] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:45 2022] DELETE /identity/v3/users/7dc0bd3be2dc4c4898aa94589a2037d5 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:45.880662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:45.881132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:45.881563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:45.906738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:45.908139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:45.933495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f3d81b4a0d24e3ea7e8dcd841aa4775'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.940593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f3d81b4a0d24e3ea7e8dcd841aa4775'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.950701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f3d81b4a0d24e3ea7e8dcd841aa4775'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.965885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f3d81b4a0d24e3ea7e8dcd841aa4775'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.966683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3749b830-18df-423d-8b5d-88c0d2440521 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f3d81b4a0d24e3ea7e8dcd841aa4775'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:45.973011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 998/1997] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:45 2022] DELETE /identity/v3/users/9f3d81b4a0d24e3ea7e8dcd841aa4775 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:45.976388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1000/1998] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:45.977432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:45.977726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:45.978006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:45.986034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:45.987449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:45.999631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:46.000850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:46.045952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a5ef9e-7c00-42bf-9ae6-592cbc6a57ec None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.066500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 999/1999] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:46.079319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:46.080820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:46.101736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.102358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.102634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.103091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.113054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.114503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:46.127583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1001/2000] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:46.150864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9276a08f-77d2-46e8-8a22-b789508fc631 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.154624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.155403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.155692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.155964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.166364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.167890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:46.178072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1000/2001] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:11:46.423722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:46.425211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:46.443687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3c14a5-989c-424b-a595-a894ec6148b7 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.449715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.450402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.450689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.450983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] PATH_INFO: `/v3/projects/0b98732cfa174357a82beef1a2af41c2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.452564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b98732cfa174357a82beef1a2af41c2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.454976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:46.499387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:46.507595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b98732cfa174357a82beef1a2af41c2', 'request_id': 'req-008c9e71-be4f-41a2-acd4-750623dd6be1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.513508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:46.515123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:46.518311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b98732cfa174357a82beef1a2af41c2', 'request_id': 'req-008c9e71-be4f-41a2-acd4-750623dd6be1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.527295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b98732cfa174357a82beef1a2af41c2', 'request_id': 'req-008c9e71-be4f-41a2-acd4-750623dd6be1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.531067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb0d32c-5a21-4466-882c-ebda5c212455 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.541929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] The token cache is being invalidate because project 0b98732cfa174357a82beef1a2af41c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:46.542351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.543391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.545250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1002/2002] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:46 2022] DELETE /identity/v3/projects/0b98732cfa174357a82beef1a2af41c2 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:46.549045 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:46.550375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:46.564177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:46.594972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.595772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.596168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.596442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] PATH_INFO: `/v3/projects/9271e6aaf1ea446888d612cf5b40000b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.597627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9271e6aaf1ea446888d612cf5b40000b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.599669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:46.615889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c9e71-be4f-41a2-acd4-750623dd6be1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.621176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.621833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.622116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.622400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] PATH_INFO: `/v3/users/3eefcb5387354918b82a3ac1e4454838` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.623622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] RBAC: Authorizing `identity:delete_user(user_id=3eefcb5387354918b82a3ac1e4454838)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.625693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:46.646719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9271e6aaf1ea446888d612cf5b40000b', 'request_id': 'req-66a1949c-6224-4ebe-a6e7-95858814ef46'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.656465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9271e6aaf1ea446888d612cf5b40000b', 'request_id': 'req-66a1949c-6224-4ebe-a6e7-95858814ef46'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.662121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9271e6aaf1ea446888d612cf5b40000b', 'request_id': 'req-66a1949c-6224-4ebe-a6e7-95858814ef46'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.673547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3eefcb5387354918b82a3ac1e4454838'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.674036 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] The token cache is being invalidate because project 9271e6aaf1ea446888d612cf5b40000b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:46.674313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.675068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-66a1949c-6224-4ebe-a6e7-95858814ef46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.680278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1001/2003] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:46 2022] DELETE /identity/v3/projects/9271e6aaf1ea446888d612cf5b40000b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:46.681705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4715ff-33b7-4ffd-bf37-7fb8bb4ae4e1 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.681988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4715ff-33b7-4ffd-bf37-7fb8bb4ae4e1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.682263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4715ff-33b7-4ffd-bf37-7fb8bb4ae4e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.685401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3eefcb5387354918b82a3ac1e4454838'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.692315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3eefcb5387354918b82a3ac1e4454838'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.700404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3eefcb5387354918b82a3ac1e4454838'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.701193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-3f2b59e3-04a8-499e-9db5-5cf934b8071c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3eefcb5387354918b82a3ac1e4454838'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:46.703632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8a4715ff-33b7-4ffd-bf37-7fb8bb4ae4e1 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:46.704912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8a4715ff-33b7-4ffd-bf37-7fb8bb4ae4e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:46.707730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1003/2004] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:46 2022] DELETE /identity/v3/users/3eefcb5387354918b82a3ac1e4454838 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:46.764398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1002/2005] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:46.772528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0dbacb-0c06-4469-aa08-19660fcf82c5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.772911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0dbacb-0c06-4469-aa08-19660fcf82c5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.773266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0dbacb-0c06-4469-aa08-19660fcf82c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.795747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ed0dbacb-0c06-4469-aa08-19660fcf82c5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:46.796709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ed0dbacb-0c06-4469-aa08-19660fcf82c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:46.804725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1004/2006] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:46.809237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4715ff-33b7-4ffd-bf37-7fb8bb4ae4e1 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:46.819226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4715ff-33b7-4ffd-bf37-7fb8bb4ae4e1 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.823869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.824497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.824782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.825063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.825782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:46.826092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.828220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:46.840008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1003/2007] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:46.844566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0dbacb-0c06-4469-aa08-19660fcf82c5 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:46.854332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0dbacb-0c06-4469-aa08-19660fcf82c5 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.862400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.863102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.863400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.863746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.864702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.866445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:46.897001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1005/2008] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:46 2022] POST /identity/v3/projects => generated 391 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:46.901707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:46.912842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e11857-d36e-4a1d-87dc-b641c1cf27ab admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:46.963785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:46.981805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:46.982756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:46.983233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:46.983570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:46.984251 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:46.986490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.004173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:47.006395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:47.029785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4cee9a-a20f-414d-8aa0-9c8cb0d8cb82 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:47.035762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:47.035944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1004/2009] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:46 2022] POST /identity/v3/users => generated 296 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:47.036753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.037035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.037311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] PATH_INFO: `/v3/projects/23609d1827aa41db8ff53c079832cf28` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.038500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23609d1827aa41db8ff53c079832cf28)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:47.040498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:47.040648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.054661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2810b-f31f-420d-b5ed-1b124fba0159 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:47.059691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:47.060657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:47.061321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.061613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.061914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.062598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:47.064282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.078993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1005/2010] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:11:47.083024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:47.093328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f075e0bc-92a4-4a65-a641-c342ff37e39b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:47.100256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23609d1827aa41db8ff53c079832cf28', 'request_id': 'req-d3dee621-8bc8-476f-b29a-a04fc7ec5082'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.110660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23609d1827aa41db8ff53c079832cf28', 'request_id': 'req-d3dee621-8bc8-476f-b29a-a04fc7ec5082'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.117965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23609d1827aa41db8ff53c079832cf28', 'request_id': 'req-d3dee621-8bc8-476f-b29a-a04fc7ec5082'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.131225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] The token cache is being invalidate because project 23609d1827aa41db8ff53c079832cf28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:47.131519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.132192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.133969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1006/2011] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:47 2022] DELETE /identity/v3/projects/23609d1827aa41db8ff53c079832cf28 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:47.137805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:47.139222 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:47.157369 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:47.157916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.158200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.158483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.159229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:47.160863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.172305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1006/2012] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:47.176912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:47.179027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:47.194659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17fda394-652b-4063-a680-a3dfc2b9b69b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:47.201077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9288945f-4f8d-48ff-936c-21468b8afd14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:47.201842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9288945f-4f8d-48ff-936c-21468b8afd14 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.202153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9288945f-4f8d-48ff-936c-21468b8afd14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.202455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9288945f-4f8d-48ff-936c-21468b8afd14 admin admin] PATH_INFO: `/v3/projects/6528e6f38f754a3bb78bab3cfc271e25/users/ca00808c1e8e430dba6bb6661c73e339/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.210146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dee621-8bc8-476f-b29a-a04fc7ec5082 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:47.216360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:47.217165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.217468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.217785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] PATH_INFO: `/v3/users/fe999e9fa73b4ba9850a7510e39fb3a1` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.218990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe999e9fa73b4ba9850a7510e39fb3a1)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:47.219493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9288945f-4f8d-48ff-936c-21468b8afd14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6528e6f38f754a3bb78bab3cfc271e25, user_id=ca00808c1e8e430dba6bb6661c73e339, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:47.220949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.222346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9288945f-4f8d-48ff-936c-21468b8afd14 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.248172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1007/2013] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:47 2022] PUT /identity/v3/projects/6528e6f38f754a3bb78bab3cfc271e25/users/ca00808c1e8e430dba6bb6661c73e339/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:47.260444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.260887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.261234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.279668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe999e9fa73b4ba9850a7510e39fb3a1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.297671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] MFA Rules not processed for user `ca00808c1e8e430dba6bb6661c73e339`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:47.297929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe999e9fa73b4ba9850a7510e39fb3a1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.299800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:47.308057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe999e9fa73b4ba9850a7510e39fb3a1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.332081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe999e9fa73b4ba9850a7510e39fb3a1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.332904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe999e9fa73b4ba9850a7510e39fb3a1'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.344674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1007/2014] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:47 2022] DELETE /identity/v3/users/fe999e9fa73b4ba9850a7510e39fb3a1 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:11:47.404479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1008/2015] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:47 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:47.413603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:47.415084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:47.442037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:47.477690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1bacf1-4c4d-489f-880a-b1252acf0c13 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:47.481359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55dc99d6-b9a1-4218-a629-b2a247807064 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:47.481992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dc99d6-b9a1-4218-a629-b2a247807064 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.482276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dc99d6-b9a1-4218-a629-b2a247807064 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.482554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55dc99d6-b9a1-4218-a629-b2a247807064 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.543958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dc99d6-b9a1-4218-a629-b2a247807064 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:47.545813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55dc99d6-b9a1-4218-a629-b2a247807064 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.562469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1008/2016] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:47 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:47.724047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:47.725590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:47.767253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:47.831357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e469da8-2ad5-4745-8aa2-5179b399b04b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:47.898279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:47.898987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.899274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.899930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] PATH_INFO: `/v3/projects/224696e362c34e0e8a6558b2e456d7c4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:47.901218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] RBAC: Authorizing `identity:delete_project(project_id=224696e362c34e0e8a6558b2e456d7c4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:47.903219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:47.960971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '224696e362c34e0e8a6558b2e456d7c4', 'request_id': 'req-213182e9-f1d5-4dc9-ae5c-4150f03db650'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.969848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '224696e362c34e0e8a6558b2e456d7c4', 'request_id': 'req-213182e9-f1d5-4dc9-ae5c-4150f03db650'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.976646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '224696e362c34e0e8a6558b2e456d7c4', 'request_id': 'req-213182e9-f1d5-4dc9-ae5c-4150f03db650'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.989500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] The token cache is being invalidate because project 224696e362c34e0e8a6558b2e456d7c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:47.989792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.990466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-213182e9-f1d5-4dc9-ae5c-4150f03db650 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:47.992342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1009/2017] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:47 2022] DELETE /identity/v3/projects/224696e362c34e0e8a6558b2e456d7c4 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:47.998053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407fd067-790a-48da-8a14-7266f92b9770 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:47.998554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407fd067-790a-48da-8a14-7266f92b9770 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:47.998630 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407fd067-790a-48da-8a14-7266f92b9770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.019161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-407fd067-790a-48da-8a14-7266f92b9770 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:48.020381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-407fd067-790a-48da-8a14-7266f92b9770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:48.079800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1009/2018] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:48.088268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b077c-9193-426c-9cb9-6c10cbcc0643 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.088561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b077c-9193-426c-9cb9-6c10cbcc0643 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.088844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b077c-9193-426c-9cb9-6c10cbcc0643 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.109211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-185b077c-9193-426c-9cb9-6c10cbcc0643 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:48.110422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-185b077c-9193-426c-9cb9-6c10cbcc0643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:48.119556 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1010/2019] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:48.123984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407fd067-790a-48da-8a14-7266f92b9770 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.134770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407fd067-790a-48da-8a14-7266f92b9770 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.141185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.141822 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.142099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.142365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.143265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:48.143831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.146776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.161557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1010/2020] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:48.168797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b077c-9193-426c-9cb9-6c10cbcc0643 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.184169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b077c-9193-426c-9cb9-6c10cbcc0643 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.189915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.190532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.190811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.191123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.191799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.193713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.224248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1011/2021] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:48 2022] POST /identity/v3/projects => generated 373 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:48.230065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.240927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39237b01-b3e9-4d13-8efe-843725efe0e1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.298747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.299405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.299683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.299956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.300625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.302531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.312694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:48.329607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1011/2022] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:48 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:11:48.331124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.341847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21040fd2-6bd4-4306-b8cf-7aded4662597 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.347894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.348514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.348820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.349092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.349793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.351475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.362215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1012/2023] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:48.366410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.379121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb68b1d-a4d2-44f1-9958-22acdaab8cd8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.382106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0b1607-e3d9-4f81-9f98-0297281bf3ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.382721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0b1607-e3d9-4f81-9f98-0297281bf3ce admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.382990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0b1607-e3d9-4f81-9f98-0297281bf3ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.383294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0b1607-e3d9-4f81-9f98-0297281bf3ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.383993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0b1607-e3d9-4f81-9f98-0297281bf3ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.385615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0b1607-e3d9-4f81-9f98-0297281bf3ce admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.397250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1012/2024] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:48.401242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.411713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e5d26a-53ae-47ac-873c-db84c152490f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.416814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.417594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.417971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.418319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] PATH_INFO: `/v3/projects/6f71eebc9ce647f9be7f78cc22c15f6c/users/65e42f4b33664fd3884ba1d9ef89d8a4/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.434207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f71eebc9ce647f9be7f78cc22c15f6c, user_id=65e42f4b33664fd3884ba1d9ef89d8a4, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.436967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.455501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1013/2025] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:48 2022] PUT /identity/v3/projects/6f71eebc9ce647f9be7f78cc22c15f6c/users/65e42f4b33664fd3884ba1d9ef89d8a4/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:48.466705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d50d5a5-6c1e-484b-ac8a-641e116c32c1 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.467014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d50d5a5-6c1e-484b-ac8a-641e116c32c1 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.467313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d50d5a5-6c1e-484b-ac8a-641e116c32c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.503520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3d50d5a5-6c1e-484b-ac8a-641e116c32c1 None None] MFA Rules not processed for user `65e42f4b33664fd3884ba1d9ef89d8a4`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:48.504800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3d50d5a5-6c1e-484b-ac8a-641e116c32c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:48.578010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1013/2026] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:48 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:48.582519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.587315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:48.647222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b019bbf5-59c1-4a33-8244-3d9686edfa15 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.652190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.652790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.653099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.653373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.654120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.656182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.684872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1014/2027] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:48 2022] POST /identity/v3/projects => generated 375 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:48.688964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d50d5a5-6c1e-484b-ac8a-641e116c32c1 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.700597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d50d5a5-6c1e-484b-ac8a-641e116c32c1 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.759326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.760237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.760594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.760945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.761984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.764930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.795702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1014/2028] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:48 2022] POST /identity/v3/users => generated 294 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:48.800344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.810837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6018a6e6-8fe5-446a-b61b-f9d0dedee03c admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.815856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.816502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.816783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.817053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.817762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.819499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.830756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1015/2029] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:48.834995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.847445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84feac87-28f2-498b-b80f-1e7db89f38da admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.856075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.857036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.857437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.857845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] PATH_INFO: `/v3/projects/d030c6827b034472ab082e97629be837/users/bcae922e21a24cf79e545f8f38c32f15/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.879041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d030c6827b034472ab082e97629be837, user_id=bcae922e21a24cf79e545f8f38c32f15, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.881741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.899360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1015/2030] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:48 2022] PUT /identity/v3/projects/d030c6827b034472ab082e97629be837/users/bcae922e21a24cf79e545f8f38c32f15/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:48.904443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:48.916367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6189e03-693a-46e7-b479-637267bd3efd admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:48.974549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec2b04-0f1f-427c-b98e-6288c4ce808c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:48.975129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec2b04-0f1f-427c-b98e-6288c4ce808c admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:48.975407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec2b04-0f1f-427c-b98e-6288c4ce808c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:48.975677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec2b04-0f1f-427c-b98e-6288c4ce808c admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:48.976383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ec2b04-0f1f-427c-b98e-6288c4ce808c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:48.977986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ec2b04-0f1f-427c-b98e-6288c4ce808c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:48.987722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1016/2031] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:48.992067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:49.002528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626f4c83-a38b-4e28-b178-6567bb76a353 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:49.007609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:49.008306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:49.008583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:49.008841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] PATH_INFO: `/v3/projects/d030c6827b034472ab082e97629be837/users/bcae922e21a24cf79e545f8f38c32f15/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:49.010873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d030c6827b034472ab082e97629be837, user_id=bcae922e21a24cf79e545f8f38c32f15, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:49.013552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:49.024172 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1016/2032] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:48 2022] PUT /identity/v3/projects/d030c6827b034472ab082e97629be837/users/bcae922e21a24cf79e545f8f38c32f15/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:49.035333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0399ee-2e18-4a68-b952-0e899e606cb2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:49.035614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0399ee-2e18-4a68-b952-0e899e606cb2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:49.035891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0399ee-2e18-4a68-b952-0e899e606cb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:49.063910 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ec0399ee-2e18-4a68-b952-0e899e606cb2 None None] MFA Rules not processed for user `bcae922e21a24cf79e545f8f38c32f15`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:49.065172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ec0399ee-2e18-4a68-b952-0e899e606cb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:49.132400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:49.142385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1017/2033] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:49 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:49.150306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:49.152294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:49.185155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:49.222406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4765be-ecaf-43a6-b87b-74c70045378a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:49.226301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e14cb0f-625b-4370-bc13-0bba49fa3c96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:49.226846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e14cb0f-625b-4370-bc13-0bba49fa3c96 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:49.227185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e14cb0f-625b-4370-bc13-0bba49fa3c96 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:49.227464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e14cb0f-625b-4370-bc13-0bba49fa3c96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:49.283922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e14cb0f-625b-4370-bc13-0bba49fa3c96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:49.285366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e14cb0f-625b-4370-bc13-0bba49fa3c96 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:49.294330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1017/2034] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:49 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:52.929267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0399ee-2e18-4a68-b952-0e899e606cb2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:52.993437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0399ee-2e18-4a68-b952-0e899e606cb2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:52.999920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:53.000834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:53.001224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:53.001615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] PATH_INFO: `/v3/users/65e42f4b33664fd3884ba1d9ef89d8a4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:53.003241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65e42f4b33664fd3884ba1d9ef89d8a4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:53.005284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:53.057327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65e42f4b33664fd3884ba1d9ef89d8a4'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.066562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65e42f4b33664fd3884ba1d9ef89d8a4'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.075748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65e42f4b33664fd3884ba1d9ef89d8a4'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.088443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65e42f4b33664fd3884ba1d9ef89d8a4'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.089244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65e42f4b33664fd3884ba1d9ef89d8a4'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.095644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1018/2035] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:52 2022] DELETE /identity/v3/users/65e42f4b33664fd3884ba1d9ef89d8a4 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:53.100266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169d012-3876-480c-b061-deb632d4693d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:53.100745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169d012-3876-480c-b061-deb632d4693d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:53.100830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169d012-3876-480c-b061-deb632d4693d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:53.127485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1169d012-3876-480c-b061-deb632d4693d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:53.129096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1169d012-3876-480c-b061-deb632d4693d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:53.202199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1018/2036] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:11:53.212437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:53.213992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:53.282982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03395d3-0971-4f92-9f22-6f09519c9587 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:53.286697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:53.287559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:53.287886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:53.288395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:53.300819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:53.302516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:53.314057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1019/2037] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:11:53.892196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169d012-3876-480c-b061-deb632d4693d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:53.893702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1169d012-3876-480c-b061-deb632d4693d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:53.905815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169d012-3876-480c-b061-deb632d4693d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:53.911597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:53.912260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:53.912531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:53.912803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] PATH_INFO: `/v3/projects/6f71eebc9ce647f9be7f78cc22c15f6c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:53.914096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f71eebc9ce647f9be7f78cc22c15f6c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:53.916273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:53.967305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f71eebc9ce647f9be7f78cc22c15f6c', 'request_id': 'req-e74c4888-99ac-4190-aeb4-687b3dea8676'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.977806 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f71eebc9ce647f9be7f78cc22c15f6c', 'request_id': 'req-e74c4888-99ac-4190-aeb4-687b3dea8676'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.984172 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f71eebc9ce647f9be7f78cc22c15f6c', 'request_id': 'req-e74c4888-99ac-4190-aeb4-687b3dea8676'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.995880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] The token cache is being invalidate because project 6f71eebc9ce647f9be7f78cc22c15f6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:53.996171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.996830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:53.998483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1019/2038] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:53 2022] DELETE /identity/v3/projects/6f71eebc9ce647f9be7f78cc22c15f6c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:54.002663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:54.012774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1078156a-b0a0-46fe-a143-6e4c24bdfae7 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:54.031205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:54.072567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:54.073221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.073516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.073791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] PATH_INFO: `/v3/users/bcae922e21a24cf79e545f8f38c32f15` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.075056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bcae922e21a24cf79e545f8f38c32f15)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:54.076987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:54.128490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bcae922e21a24cf79e545f8f38c32f15'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.135550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bcae922e21a24cf79e545f8f38c32f15'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.145930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bcae922e21a24cf79e545f8f38c32f15'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.158532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bcae922e21a24cf79e545f8f38c32f15'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.159279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bc0b500b-3fff-4576-8ed5-17083a8e3c15 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bcae922e21a24cf79e545f8f38c32f15'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.165674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1020/2039] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:11:54 2022] DELETE /identity/v3/users/bcae922e21a24cf79e545f8f38c32f15 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:54.490248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:54.491888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:54.500280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:54.506525 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74c4888-99ac-4190-aeb4-687b3dea8676 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:54.572824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:54.573489 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.573788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.574061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] PATH_INFO: `/v3/projects/d030c6827b034472ab082e97629be837` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.575289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d030c6827b034472ab082e97629be837)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:54.577193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:54.624754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd030c6827b034472ab082e97629be837', 'request_id': 'req-e1617008-3650-42fb-afc8-a592379cc8c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.633354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd030c6827b034472ab082e97629be837', 'request_id': 'req-e1617008-3650-42fb-afc8-a592379cc8c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.639575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd030c6827b034472ab082e97629be837', 'request_id': 'req-e1617008-3650-42fb-afc8-a592379cc8c2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.651339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] The token cache is being invalidate because project d030c6827b034472ab082e97629be837 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:11:54.651602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.652306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e1617008-3650-42fb-afc8-a592379cc8c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:11:54.653933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1020/2040] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:11:54 2022] DELETE /identity/v3/projects/d030c6827b034472ab082e97629be837 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:54.659053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c808f0-779c-4cdc-ae34-6913d852bf27 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.659341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c808f0-779c-4cdc-ae34-6913d852bf27 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.659628 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c808f0-779c-4cdc-ae34-6913d852bf27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.680934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-56c808f0-779c-4cdc-ae34-6913d852bf27 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:54.682159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-56c808f0-779c-4cdc-ae34-6913d852bf27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:54.746082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1021/2041] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:54.754135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d71932-9bad-4d62-a27f-b6f4181ad290 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.754423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d71932-9bad-4d62-a27f-b6f4181ad290 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.754718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d71932-9bad-4d62-a27f-b6f4181ad290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.776286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-05d71932-9bad-4d62-a27f-b6f4181ad290 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:54.777733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-05d71932-9bad-4d62-a27f-b6f4181ad290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:54.788272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1021/2042] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:54.793198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c808f0-779c-4cdc-ae34-6913d852bf27 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:54.803156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c808f0-779c-4cdc-ae34-6913d852bf27 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:54.808631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:54.809254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.809536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.809817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.810524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:11:54.810827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:54.812605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:54.825299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1022/2043] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:11:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:11:54.830559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d71932-9bad-4d62-a27f-b6f4181ad290 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:54.840171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d71932-9bad-4d62-a27f-b6f4181ad290 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:54.845207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:54.845834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.846117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.846394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.847115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:54.849164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:54.879027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1022/2044] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:11:54 2022] POST /identity/v3/projects => generated 375 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:54.882298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:54.892600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e464184-3f39-4604-905c-de6ffa56a094 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:54.916744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:54.950577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:54.951255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.951538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.951836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.952515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:54.955023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:54.977707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1023/2045] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:11:54 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:11:54.982104 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:54.992605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be0a63f-d745-4e11-8fe0-878b480570bb admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:54.997977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:54.998608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:54.998888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:54.999190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:54.999981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:55.001589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:55.012758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1023/2046] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:55.017022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:55.027965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691cb336-d0bb-4435-88a4-c0c83f20a0e5 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:55.033428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b627598f-1f34-4ae6-be29-15adc6e9a40b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:55.034057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b627598f-1f34-4ae6-be29-15adc6e9a40b admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:55.034344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b627598f-1f34-4ae6-be29-15adc6e9a40b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:55.034610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b627598f-1f34-4ae6-be29-15adc6e9a40b admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:55.035358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b627598f-1f34-4ae6-be29-15adc6e9a40b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:55.036976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b627598f-1f34-4ae6-be29-15adc6e9a40b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:55.047865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1024/2047] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:11:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:11:55.052059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:55.062742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115b25c3-eda2-426f-924c-a592db2ae112 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:55.067948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:55.068617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:55.068929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:55.069205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] PATH_INFO: `/v3/projects/77c81603592f4e41b1496d29a226ec8c/users/d8bfa0a52b3d4792b3fba7275357e229/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:55.084316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77c81603592f4e41b1496d29a226ec8c, user_id=d8bfa0a52b3d4792b3fba7275357e229, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:55.086981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:55.101352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1024/2048] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:11:55 2022] PUT /identity/v3/projects/77c81603592f4e41b1496d29a226ec8c/users/d8bfa0a52b3d4792b3fba7275357e229/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:11:55.112024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb39cb49-9975-4f7b-80ca-c92735669654 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:55.112388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb39cb49-9975-4f7b-80ca-c92735669654 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:55.112680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb39cb49-9975-4f7b-80ca-c92735669654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:55.140249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-cb39cb49-9975-4f7b-80ca-c92735669654 None None] MFA Rules not processed for user `d8bfa0a52b3d4792b3fba7275357e229`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:11:55.141503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cb39cb49-9975-4f7b-80ca-c92735669654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:55.206787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1025/2049] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:11:55 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:11:55.221000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:11:55.223146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:11:55.256708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:11:55.283795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb770d9-cbd4-425b-b245-f980875f1fa8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:11:55.287909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6867a8-61a9-43e6-ad4f-380e170eb6be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:11:55.288534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6867a8-61a9-43e6-ad4f-380e170eb6be service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:11:55.288818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6867a8-61a9-43e6-ad4f-380e170eb6be service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:11:55.289095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6867a8-61a9-43e6-ad4f-380e170eb6be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:11:55.297778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6867a8-61a9-43e6-ad4f-380e170eb6be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:11:55.299249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6867a8-61a9-43e6-ad4f-380e170eb6be service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:11:55.308027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1025/2050] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:11:55 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:01.655321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb39cb49-9975-4f7b-80ca-c92735669654 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:01.726854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb39cb49-9975-4f7b-80ca-c92735669654 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:01.731393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:01.732041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:01.732320 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:01.732594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] PATH_INFO: `/v3/users/d8bfa0a52b3d4792b3fba7275357e229` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:01.733753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8bfa0a52b3d4792b3fba7275357e229)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:01.735718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:01.779382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:01.790465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8bfa0a52b3d4792b3fba7275357e229'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:01.797858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8bfa0a52b3d4792b3fba7275357e229'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:01.806787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8bfa0a52b3d4792b3fba7275357e229'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:01.821757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8bfa0a52b3d4792b3fba7275357e229'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:01.822475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8bfa0a52b3d4792b3fba7275357e229'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:01.830181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1026/2051] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:01 2022] DELETE /identity/v3/users/d8bfa0a52b3d4792b3fba7275357e229 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:01.834875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:01.835422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:01.835515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:01.856234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:01.857846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:01.920017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1026/2052] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:01.934759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:01.937167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:01.998390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd030abc-1260-4842-8e55-d06f4deaf69f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.002683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd48bdc-1a12-4d8b-b2ac-e984cc4e06d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.003322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd48bdc-1a12-4d8b-b2ac-e984cc4e06d9 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.003675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd48bdc-1a12-4d8b-b2ac-e984cc4e06d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.003961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd48bdc-1a12-4d8b-b2ac-e984cc4e06d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.017923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd48bdc-1a12-4d8b-b2ac-e984cc4e06d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.020099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd48bdc-1a12-4d8b-b2ac-e984cc4e06d9 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.032716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1027/2053] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:02.314637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:02.316141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:02.328887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ad52af-41d1-493a-9a50-29c0224ba5eb None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.335362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.336026 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.336449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.336817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] PATH_INFO: `/v3/projects/77c81603592f4e41b1496d29a226ec8c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.338187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77c81603592f4e41b1496d29a226ec8c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.340306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.398044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77c81603592f4e41b1496d29a226ec8c', 'request_id': 'req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:02.408276 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77c81603592f4e41b1496d29a226ec8c', 'request_id': 'req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:02.414516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77c81603592f4e41b1496d29a226ec8c', 'request_id': 'req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:02.426746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] The token cache is being invalidate because project 77c81603592f4e41b1496d29a226ec8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:02.427003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:02.427655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0136fc48-9ecc-4a98-bf3a-57ebac14fbc4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:02.429169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1027/2054] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:02 2022] DELETE /identity/v3/projects/77c81603592f4e41b1496d29a226ec8c => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:02.434064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d01e7c-168b-4657-b957-d7ae9dcd2f68 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.434419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d01e7c-168b-4657-b957-d7ae9dcd2f68 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.434785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d01e7c-168b-4657-b957-d7ae9dcd2f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.455609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-17d01e7c-168b-4657-b957-d7ae9dcd2f68 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:02.457003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-17d01e7c-168b-4657-b957-d7ae9dcd2f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:02.524514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1028/2055] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:02.595112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837e62e2-9d2d-46d8-8379-03b00a84cf86 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.595112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837e62e2-9d2d-46d8-8379-03b00a84cf86 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.595112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837e62e2-9d2d-46d8-8379-03b00a84cf86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.615343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-837e62e2-9d2d-46d8-8379-03b00a84cf86 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:02.616549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-837e62e2-9d2d-46d8-8379-03b00a84cf86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:02.624820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1028/2056] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:02.629478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d01e7c-168b-4657-b957-d7ae9dcd2f68 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:02.641665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d01e7c-168b-4657-b957-d7ae9dcd2f68 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.647160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.647830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.648111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.648387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.649094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:02.649432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.651065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.663094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1029/2057] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:12:02.667448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837e62e2-9d2d-46d8-8379-03b00a84cf86 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:02.677038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837e62e2-9d2d-46d8-8379-03b00a84cf86 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.681810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.682422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.682694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.682979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.683682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.685608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.715306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1029/2058] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:02 2022] POST /identity/v3/projects => generated 375 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:02.719613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:02.730075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fafd13-1ac2-49aa-bfab-98e9cde8e278 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.775870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:02.793837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.794464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.794748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.795053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.795723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.797675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.827331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1030/2059] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:02 2022] POST /identity/v3/users => generated 288 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:02.832041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:02.842569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227a80dd-d629-490f-8d97-42511e2c5176 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.848200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.848811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.849089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.849355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.850043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.851686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.861988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1030/2060] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:02.865964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:02.876359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf11155-3b2b-4395-b8b4-c4ffd18538cd admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.881703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c92cf63-eabd-4212-aa95-204b504cc03d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.882323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c92cf63-eabd-4212-aa95-204b504cc03d admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.882606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c92cf63-eabd-4212-aa95-204b504cc03d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.882864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c92cf63-eabd-4212-aa95-204b504cc03d admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.883603 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c92cf63-eabd-4212-aa95-204b504cc03d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.885218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c92cf63-eabd-4212-aa95-204b504cc03d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.897431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1031/2061] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:02.901604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:02.912239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf2541e-b925-4de6-9e8d-f29c6c2da221 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:02.917278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:02.917950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.918223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.918499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] PATH_INFO: `/v3/projects/3c8d6492e3aa4191bf1554213bb749e9/users/f35bffdd694f4df7befe85d401191ff0/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.931196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:02.934238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c8d6492e3aa4191bf1554213bb749e9, user_id=f35bffdd694f4df7befe85d401191ff0, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:02.937295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:02.953152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1031/2062] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:02 2022] PUT /identity/v3/projects/3c8d6492e3aa4191bf1554213bb749e9/users/f35bffdd694f4df7befe85d401191ff0/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:02.963619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f0ff2-c101-491d-8f1d-9724397e34dc None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:02.963899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f0ff2-c101-491d-8f1d-9724397e34dc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:02.964167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462f0ff2-c101-491d-8f1d-9724397e34dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:02.994134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-462f0ff2-c101-491d-8f1d-9724397e34dc None None] MFA Rules not processed for user `f35bffdd694f4df7befe85d401191ff0`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:02.995433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-462f0ff2-c101-491d-8f1d-9724397e34dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:03.076349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1032/2063] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:02 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:03.088010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:03.089751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:03.147573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f907fef1-65a9-4afd-9ea0-2a324b5cdba8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:03.151349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:03.151972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:03.152252 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:03.152523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:03.160823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:03.162242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:03.170767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1032/2064] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:03 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:05.602296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca23b612-d5dd-45a0-8117-37f222353b10 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:05.602796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca23b612-d5dd-45a0-8117-37f222353b10 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:05.602978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca23b612-d5dd-45a0-8117-37f222353b10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:05.622179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ca23b612-d5dd-45a0-8117-37f222353b10 None None] MFA Rules not processed for user `f35bffdd694f4df7befe85d401191ff0`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:05.623426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ca23b612-d5dd-45a0-8117-37f222353b10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:05.630805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1033/2065] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:05 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:09.954053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:09.955786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:10.046013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c536cfbf-831c-4bea-96b8-0a93b9802aa0 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:10.051614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:10.051785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:10.052105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:10.052453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] PATH_INFO: `/v3/users/adcae81584cc447db8d2628ff6e1bf5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:10.054090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=adcae81584cc447db8d2628ff6e1bf5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:10.056207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:10.116472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'adcae81584cc447db8d2628ff6e1bf5b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.131073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'adcae81584cc447db8d2628ff6e1bf5b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.139297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'adcae81584cc447db8d2628ff6e1bf5b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.148549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'adcae81584cc447db8d2628ff6e1bf5b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.149298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-63a7d43d-0175-4ac5-a329-f71f0a7ca7e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'adcae81584cc447db8d2628ff6e1bf5b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.156135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1033/2066] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:09 2022] DELETE /identity/v3/users/adcae81584cc447db8d2628ff6e1bf5b => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:10.353720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca23b612-d5dd-45a0-8117-37f222353b10 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:10.371360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca23b612-d5dd-45a0-8117-37f222353b10 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:10.446452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:10.447226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:10.447492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:10.447850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] PATH_INFO: `/v3/projects/e5b0edb065564485bebea31ef6f20b7e` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:10.449826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5b0edb065564485bebea31ef6f20b7e)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:10.451854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:10.505168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:10.506649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5b0edb065564485bebea31ef6f20b7e', 'request_id': 'req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.517218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5b0edb065564485bebea31ef6f20b7e', 'request_id': 'req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.524212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5b0edb065564485bebea31ef6f20b7e', 'request_id': 'req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.535713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] The token cache is being invalidate because project e5b0edb065564485bebea31ef6f20b7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:10.536003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.536692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0452bd1e-5896-4db0-a4c0-3cad9ff96d01 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:10.538277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1034/2067] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:10 2022] DELETE /identity/v3/projects/e5b0edb065564485bebea31ef6f20b7e => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:10.549427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284d7299-5525-432e-9025-9f792af31f95 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:10.549625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284d7299-5525-432e-9025-9f792af31f95 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:10.549911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284d7299-5525-432e-9025-9f792af31f95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:10.570762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-284d7299-5525-432e-9025-9f792af31f95 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:10.572407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-284d7299-5525-432e-9025-9f792af31f95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:10.582176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:10.643275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1034/2068] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:10.652148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0aa419-ba1f-4ba6-b6a1-42bf9ff00c49 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:10.652566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0aa419-ba1f-4ba6-b6a1-42bf9ff00c49 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:10.652937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0aa419-ba1f-4ba6-b6a1-42bf9ff00c49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:10.677239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-5d0aa419-ba1f-4ba6-b6a1-42bf9ff00c49 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:10.678784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5d0aa419-ba1f-4ba6-b6a1-42bf9ff00c49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:10.686989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1035/2069] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:10.691697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284d7299-5525-432e-9025-9f792af31f95 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:10.706803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284d7299-5525-432e-9025-9f792af31f95 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:10.711892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:10.712582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:10.712889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:10.713184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:10.713930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:10.714291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:10.716411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:10.739604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1035/2070] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:12:10.744399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0aa419-ba1f-4ba6-b6a1-42bf9ff00c49 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:10.755684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0aa419-ba1f-4ba6-b6a1-42bf9ff00c49 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:10.760443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:10.761099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:10.761348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:10.761634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:10.762440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:10.764431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:10.809152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1036/2071] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:10 2022] POST /identity/v3/projects => generated 387 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:12:10.812638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:10.831140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212a5ba-233b-4ae6-a1f1-589b850d14f9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:10.947137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:10.948663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:10.949047 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:10.949417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:10.950204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:10.953015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:10.974404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:10.976620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:10.993418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92f805-eb42-4f66-84a9-b13285bb80ec admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:10.997651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1036/2072] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:10 2022] POST /identity/v3/users => generated 294 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:10.998855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.009392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.010292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.010423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.010642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] PATH_INFO: `/v3/users/ca00808c1e8e430dba6bb6661c73e339` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.011999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca00808c1e8e430dba6bb6661c73e339)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.012334 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b679bc16-d3c4-4c8e-b436-120758637169 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.016417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.018391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.019177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.019700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.020194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.021033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.022852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.039012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1037/2073] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:10 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:12:11.043393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.056131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc91605-047e-4745-a7b1-92ae64828d29 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.073364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.077804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.078200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.078555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.079388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.081423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.083436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca00808c1e8e430dba6bb6661c73e339'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:11.093561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca00808c1e8e430dba6bb6661c73e339'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:11.100904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1038/2074] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:11 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:11.106466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca00808c1e8e430dba6bb6661c73e339'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:11.106636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.123091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca00808c1e8e430dba6bb6661c73e339'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:11.124229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-87952b45-c84f-47f9-91d1-d46261f6551a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca00808c1e8e430dba6bb6661c73e339'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:11.132800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f857a796-afe7-4619-9912-71c3af367449 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.135312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1037/2075] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:10 2022] DELETE /identity/v3/users/ca00808c1e8e430dba6bb6661c73e339 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:11.140186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.140643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.141111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.173333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:11.174655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:11.221565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd698b9-1183-4092-9e59-c2b38913e811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.222686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd698b9-1183-4092-9e59-c2b38913e811 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.223184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd698b9-1183-4092-9e59-c2b38913e811 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.223640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd698b9-1183-4092-9e59-c2b38913e811 admin admin] PATH_INFO: `/v3/projects/aad2c3bdd1674901b25da9992498285f/users/73e7e199114e492382e65a2470b6a1b2/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.240653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:11.245621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd698b9-1183-4092-9e59-c2b38913e811 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aad2c3bdd1674901b25da9992498285f, user_id=73e7e199114e492382e65a2470b6a1b2, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.252446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd698b9-1183-4092-9e59-c2b38913e811 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.270986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1039/2076] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:11 2022] PUT /identity/v3/projects/aad2c3bdd1674901b25da9992498285f/users/73e7e199114e492382e65a2470b6a1b2/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:11.280815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.281211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.281549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.285814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1038/2077] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:11.297554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.299048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:11.348885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] MFA Rules not processed for user `73e7e199114e492382e65a2470b6a1b2`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:11.362810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:11.364276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=84,keystone:INSERT=3,keystone:DELETE=13 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:11.444683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3254659e-31ed-468a-a863-38a9ee3bc522 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.480387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1040/2078] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:11 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:11.495656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.498675 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:11.556089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.556861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.557262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.557641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.658020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cecb3be-74a9-4cf0-a36e-8bf57bf5467a None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.666725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.668396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.668571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.668688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.668830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.670129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.670314 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.677868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.699115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1039/2079] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:11.717973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1041/2080] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:11 2022] POST /identity/v3/projects => generated 385 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:11.721862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.732470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac0958a-8fce-4ed5-9702-9660b5e354d5 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.803821 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.804457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.804743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.805080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.806091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.807761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.834627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1040/2081] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:11 2022] POST /identity/v3/users => generated 299 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:11.836255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.848058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6072f5-5707-49c7-a5fc-881b23bfb8a4 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.854441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.855125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.855411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.855810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.856534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.858160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.873140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1042/2082] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:11.874440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.888390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbaa7b63-f8b0-4c19-ac6e-37446bae61c4 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:11.897626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:11.898845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:11.899258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:11.899634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] PATH_INFO: `/v3/projects/2de16157e63744a6877172ecb53295a5/users/f0df990b56974f0dafe0eeec27f04e1f/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:11.912967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:11.916634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=2de16157e63744a6877172ecb53295a5, user_id=f0df990b56974f0dafe0eeec27f04e1f, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:11.919530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:11.938788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1041/2083] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:11 2022] PUT /identity/v3/projects/2de16157e63744a6877172ecb53295a5/users/f0df990b56974f0dafe0eeec27f04e1f/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:11.942693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:11.954646 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a05a63e-6c29-4580-864c-db35856a8901 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:12.041687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:12.051613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a67f21b-b39f-4cc8-b907-5d2d0bdc0622 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:12.052826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67f21b-b39f-4cc8-b907-5d2d0bdc0622 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.053352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67f21b-b39f-4cc8-b907-5d2d0bdc0622 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.053952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a67f21b-b39f-4cc8-b907-5d2d0bdc0622 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.054920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a67f21b-b39f-4cc8-b907-5d2d0bdc0622 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:12.057015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a67f21b-b39f-4cc8-b907-5d2d0bdc0622 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:12.071008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1043/2084] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:11 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:12.072426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:12.083637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc83dbc-25b1-4538-9c6a-dca09fe015ed admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:12.088609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:12.089371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.089718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.090082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] PATH_INFO: `/v3/projects/2de16157e63744a6877172ecb53295a5/users/f0df990b56974f0dafe0eeec27f04e1f/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.092145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] RBAC: Authorizing `identity:create_grant(project_id=2de16157e63744a6877172ecb53295a5, user_id=f0df990b56974f0dafe0eeec27f04e1f, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:12.094894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:12.108018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1042/2085] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:12 2022] PUT /identity/v3/projects/2de16157e63744a6877172ecb53295a5/users/f0df990b56974f0dafe0eeec27f04e1f/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:12.120211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.120498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.120774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.150805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] MFA Rules not processed for user `f0df990b56974f0dafe0eeec27f04e1f`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:12.152085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:12.236954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1044/2086] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:12 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:12.242804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:12.244604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:12.250855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:12.252759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:12.318770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed271ab-7c36-4493-93f9-bdca6688b3d0 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:12.322596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:12.323340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.323693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.324049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.325210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5044f3dd-4bd8-47a1-9639-398b09890cec admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:12.332450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:12.333398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.333687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.334001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] PATH_INFO: `/v3/projects/6528e6f38f754a3bb78bab3cfc271e25` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.334592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:12.335445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6528e6f38f754a3bb78bab3cfc271e25)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:12.336089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:12.339883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:12.347151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1045/2087] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:12 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:12.385066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:12.391463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6528e6f38f754a3bb78bab3cfc271e25', 'request_id': 'req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.396684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dd9a6a-e8c1-4719-8ce0-2d4b3b674c2a service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:12.400540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6528e6f38f754a3bb78bab3cfc271e25', 'request_id': 'req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.407710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6528e6f38f754a3bb78bab3cfc271e25', 'request_id': 'req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.421132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] The token cache is being invalidate because project 6528e6f38f754a3bb78bab3cfc271e25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:12.421580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.422346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6596a88c-eb7b-402e-9e5e-1ff1b41229d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.435502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1043/2088] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:12 2022] DELETE /identity/v3/projects/6528e6f38f754a3bb78bab3cfc271e25 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:12.475456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:12.476243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.476591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.476938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] PATH_INFO: `/v3/users/73e7e199114e492382e65a2470b6a1b2` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.478375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73e7e199114e492382e65a2470b6a1b2)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:12.480769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:12.509087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:12.549488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73e7e199114e492382e65a2470b6a1b2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.564411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73e7e199114e492382e65a2470b6a1b2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.572767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73e7e199114e492382e65a2470b6a1b2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.583838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73e7e199114e492382e65a2470b6a1b2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.584695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73e7e199114e492382e65a2470b6a1b2'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:12.596148 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1046/2089] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:12 2022] DELETE /identity/v3/users/73e7e199114e492382e65a2470b6a1b2 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:12.598114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.598725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.599479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.639829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:12.642597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:12.721368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1044/2090] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:12.731140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:12.732620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:12.800177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b277277-3ac9-4494-b1fe-0985831e8935 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:12.804653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:12.805281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:12.805561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:12.805836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:12.815765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:12.817376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:12.828249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1047/2091] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:13.225123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:13.230052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:13.242290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f73dccb-bb8e-4960-a549-e853a7430841 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:13.248239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:13.249886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:13.249886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:13.249886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] PATH_INFO: `/v3/projects/aad2c3bdd1674901b25da9992498285f` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:13.251871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aad2c3bdd1674901b25da9992498285f)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:13.254513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:13.257716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:13.322125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aad2c3bdd1674901b25da9992498285f', 'request_id': 'req-dee8512f-0d18-4ecc-b895-dbf2d337cee0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.331436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aad2c3bdd1674901b25da9992498285f', 'request_id': 'req-dee8512f-0d18-4ecc-b895-dbf2d337cee0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.338442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aad2c3bdd1674901b25da9992498285f', 'request_id': 'req-dee8512f-0d18-4ecc-b895-dbf2d337cee0'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.350529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] The token cache is being invalidate because project aad2c3bdd1674901b25da9992498285f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:13.350814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.351495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.353377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1045/2092] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:13 2022] DELETE /identity/v3/projects/aad2c3bdd1674901b25da9992498285f => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:13.357152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:13.367664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a798256-3f5f-4440-b5d7-93b2ba635595 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:13.433582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:13.434241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:13.434518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:13.434789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] PATH_INFO: `/v3/users/f0df990b56974f0dafe0eeec27f04e1f` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:13.435966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0df990b56974f0dafe0eeec27f04e1f)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:13.437903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:13.488827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:13.491763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0df990b56974f0dafe0eeec27f04e1f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.519085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0df990b56974f0dafe0eeec27f04e1f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.530173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0df990b56974f0dafe0eeec27f04e1f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.540431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0df990b56974f0dafe0eeec27f04e1f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.541198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-50acd3c2-0ad9-4a5a-8760-ae9329e47d4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0df990b56974f0dafe0eeec27f04e1f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:13.558686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1048/2093] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:13 2022] DELETE /identity/v3/users/f0df990b56974f0dafe0eeec27f04e1f => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:13.926420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:13.928033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:13.955742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee8512f-0d18-4ecc-b895-dbf2d337cee0 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.016735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.017076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.017361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.017640 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] PATH_INFO: `/v3/projects/2de16157e63744a6877172ecb53295a5` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.018852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2de16157e63744a6877172ecb53295a5)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.020806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.072077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2de16157e63744a6877172ecb53295a5', 'request_id': 'req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:14.080688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2de16157e63744a6877172ecb53295a5', 'request_id': 'req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:14.087569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2de16157e63744a6877172ecb53295a5', 'request_id': 'req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:14.099989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] The token cache is being invalidate because project 2de16157e63744a6877172ecb53295a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:14.100283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:14.101283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-be89c559-a5cc-43dd-82b8-b1ba5caa76a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:14.102975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1046/2094] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:13 2022] DELETE /identity/v3/projects/2de16157e63744a6877172ecb53295a5 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:14.107906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e25b4c-663d-4b86-bc2e-e209aa9c11a1 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.108316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e25b4c-663d-4b86-bc2e-e209aa9c11a1 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.108692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e25b4c-663d-4b86-bc2e-e209aa9c11a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.142760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-37e25b4c-663d-4b86-bc2e-e209aa9c11a1 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:14.144645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-37e25b4c-663d-4b86-bc2e-e209aa9c11a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:14.206899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1049/2095] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:14.214874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caece665-883a-496e-913c-012c2aa27e79 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.215300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caece665-883a-496e-913c-012c2aa27e79 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.215595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caece665-883a-496e-913c-012c2aa27e79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.237494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-caece665-883a-496e-913c-012c2aa27e79 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:14.238756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-caece665-883a-496e-913c-012c2aa27e79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:14.247679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1047/2096] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:14.252137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e25b4c-663d-4b86-bc2e-e209aa9c11a1 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.262968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e25b4c-663d-4b86-bc2e-e209aa9c11a1 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.268099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.268728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.269006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.269315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.270023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:14.270333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.272147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.284257 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1050/2097] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:12:14.288599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caece665-883a-496e-913c-012c2aa27e79 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.298418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caece665-883a-496e-913c-012c2aa27e79 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.304066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.304725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.305009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.305307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.305972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.308008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.323923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:14.339024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1048/2098] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:14 2022] POST /identity/v3/projects => generated 401 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:14.344507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.355374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4386099f-6fd1-4c4c-9897-fbe885f3daa9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.412573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.413279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.413443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.413745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.414423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.416393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.441945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1051/2099] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:14 2022] POST /identity/v3/users => generated 301 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:12:14.443848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.455219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2917555a-6773-44e8-85a1-a7e6b959cefc admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.460335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.460957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.461242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.461511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.462221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.463876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.475141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1049/2100] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:14.479351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.489971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90d0dda-6c4d-4adb-8611-9571c2a0832c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.495379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f7f4c6-76ac-4b4d-bb01-cf8a103446c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.496006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f7f4c6-76ac-4b4d-bb01-cf8a103446c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.496314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f7f4c6-76ac-4b4d-bb01-cf8a103446c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.496562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f7f4c6-76ac-4b4d-bb01-cf8a103446c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.497269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f7f4c6-76ac-4b4d-bb01-cf8a103446c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.499247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f7f4c6-76ac-4b4d-bb01-cf8a103446c4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.511453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1052/2101] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:14.516076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.527062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eba0e1f-1a08-4ff1-ab91-008e61fba296 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.532862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.533526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.533812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.534086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] PATH_INFO: `/v3/projects/77758e650e544b9f891b06095170417c/users/65216926e2354ec7afc96facefb55d53/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.549478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77758e650e544b9f891b06095170417c, user_id=65216926e2354ec7afc96facefb55d53, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.552231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.568424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1050/2102] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:14 2022] PUT /identity/v3/projects/77758e650e544b9f891b06095170417c/users/65216926e2354ec7afc96facefb55d53/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:14.579458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f622c17-abe6-4364-9f09-d55728deb39f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.579756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f622c17-abe6-4364-9f09-d55728deb39f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.580030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f622c17-abe6-4364-9f09-d55728deb39f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.608575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-9f622c17-abe6-4364-9f09-d55728deb39f None None] MFA Rules not processed for user `65216926e2354ec7afc96facefb55d53`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:14.609813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9f622c17-abe6-4364-9f09-d55728deb39f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:14.622637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:14.689173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1053/2103] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:14 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:14.690421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.758874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ea07f4-5f34-48ec-91ee-139fad498867 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.763801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.764420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.764714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.764998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.765669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.767636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.801797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1051/2104] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:14 2022] POST /identity/v3/projects => generated 399 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:14.803068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f622c17-abe6-4364-9f09-d55728deb39f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.814762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f622c17-abe6-4364-9f09-d55728deb39f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.877225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.877868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.878140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.878408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.879139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.881115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.907665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1054/2105] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:14 2022] POST /identity/v3/users => generated 306 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:14.908955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.919442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3442427d-b332-4e17-a5af-a5e48900435e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.924263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.924887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.925169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.925444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.926129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.927842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.939752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1052/2106] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:12:14.943820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:14.957433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e6b0b9-b305-43e2-ab37-bf4148c71666 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:14.964672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:14.964672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:14.964672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:14.964942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] PATH_INFO: `/v3/projects/e1cf25c107da47a5827d4e12f1625521/users/657fc5b1bd2b431f8a38d905bc964c2c/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:14.980458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1cf25c107da47a5827d4e12f1625521, user_id=657fc5b1bd2b431f8a38d905bc964c2c, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:14.983204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:14.997990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1055/2107] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:14 2022] PUT /identity/v3/projects/e1cf25c107da47a5827d4e12f1625521/users/657fc5b1bd2b431f8a38d905bc964c2c/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:15.002167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:15.012936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f492e84-d185-4603-a11e-a0d070bf1306 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:15.073679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc8b469-19ec-40ec-9508-970f955b3a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:15.074311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8b469-19ec-40ec-9508-970f955b3a59 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:15.074590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8b469-19ec-40ec-9508-970f955b3a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:15.074892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc8b469-19ec-40ec-9508-970f955b3a59 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:15.075633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc8b469-19ec-40ec-9508-970f955b3a59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:15.077583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc8b469-19ec-40ec-9508-970f955b3a59 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:15.088855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1053/2108] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:15 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:15.093364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:15.104795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cc7c2a-dbba-4aaf-978c-2bcac578c536 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:15.109542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:15.110215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:15.110495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:15.110774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] PATH_INFO: `/v3/projects/e1cf25c107da47a5827d4e12f1625521/users/657fc5b1bd2b431f8a38d905bc964c2c/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:15.112815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1cf25c107da47a5827d4e12f1625521, user_id=657fc5b1bd2b431f8a38d905bc964c2c, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:15.115635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:15.126020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1056/2109] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:15 2022] PUT /identity/v3/projects/e1cf25c107da47a5827d4e12f1625521/users/657fc5b1bd2b431f8a38d905bc964c2c/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:15.137546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f620b67-324b-4ea5-aa24-1888da316573 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:15.137805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f620b67-324b-4ea5-aa24-1888da316573 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:15.138110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f620b67-324b-4ea5-aa24-1888da316573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:15.166793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6f620b67-324b-4ea5-aa24-1888da316573 None None] MFA Rules not processed for user `657fc5b1bd2b431f8a38d905bc964c2c`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:15.168144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6f620b67-324b-4ea5-aa24-1888da316573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:15.186384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:15.270461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1054/2110] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:15 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:15.282243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:15.284040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:15.350184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb36078-709e-4f4d-8d02-20362dc5f764 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:15.354450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:15.355136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:15.355417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:15.355693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:15.365828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:15.367320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:15.377065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1057/2111] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:15 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:15.457967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f620b67-324b-4ea5-aa24-1888da316573 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:15.472708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f620b67-324b-4ea5-aa24-1888da316573 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:15.476947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948b55fb-8a82-4ed6-8f4f-73b17bd9b22d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:15.477565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948b55fb-8a82-4ed6-8f4f-73b17bd9b22d service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:15.477846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948b55fb-8a82-4ed6-8f4f-73b17bd9b22d service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:15.478122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948b55fb-8a82-4ed6-8f4f-73b17bd9b22d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:15.537006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948b55fb-8a82-4ed6-8f4f-73b17bd9b22d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:15.538478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948b55fb-8a82-4ed6-8f4f-73b17bd9b22d service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:15.548238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1055/2112] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:15 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:19.708698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:19.721993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:19.777384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587ac7b0-3c29-444e-8b8a-e3caecd1c64e service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:19.782039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:19.782699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:19.783011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:19.783289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] PATH_INFO: `/v3/users/65216926e2354ec7afc96facefb55d53` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:19.784492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65216926e2354ec7afc96facefb55d53)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:19.786446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:19.843842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65216926e2354ec7afc96facefb55d53'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:19.856952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65216926e2354ec7afc96facefb55d53'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:19.863622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65216926e2354ec7afc96facefb55d53'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:19.874581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65216926e2354ec7afc96facefb55d53'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:19.875313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65216926e2354ec7afc96facefb55d53'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:19.881691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1058/2113] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:19 2022] DELETE /identity/v3/users/65216926e2354ec7afc96facefb55d53 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:19.886304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:19.886777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:19.886877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:19.906015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:19.907265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:19.987045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1056/2114] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:20.000514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:20.001382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:20.065425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d485af92-15a6-40ff-855f-67a264e92d08 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:20.071679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:20.072356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:20.072639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:20.072916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:20.085365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:20.086865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:20.097530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1059/2115] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:20.398389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:20.400031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:20.413664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff8c5f5-43cb-4fb9-91ed-f62ce14218d2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:20.420504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:20.421280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:20.421541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:20.421920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] PATH_INFO: `/v3/projects/77758e650e544b9f891b06095170417c` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:20.424295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] RBAC: Authorizing `identity:delete_project(project_id=77758e650e544b9f891b06095170417c)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:20.426812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:20.476745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:20.477996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77758e650e544b9f891b06095170417c', 'request_id': 'req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.486740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77758e650e544b9f891b06095170417c', 'request_id': 'req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.492743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77758e650e544b9f891b06095170417c', 'request_id': 'req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.504336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] The token cache is being invalidate because project 77758e650e544b9f891b06095170417c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:20.504627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.505268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.506810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1057/2116] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:20 2022] DELETE /identity/v3/projects/77758e650e544b9f891b06095170417c => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:20.511833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:20.527196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0502b75d-991b-4a93-841d-7ec6cd634b49 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:20.564110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:20.593554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:20.594200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:20.594481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:20.594754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] PATH_INFO: `/v3/users/657fc5b1bd2b431f8a38d905bc964c2c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:20.595950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=657fc5b1bd2b431f8a38d905bc964c2c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:20.597881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:20.644230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '657fc5b1bd2b431f8a38d905bc964c2c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.656632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '657fc5b1bd2b431f8a38d905bc964c2c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.663785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '657fc5b1bd2b431f8a38d905bc964c2c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.672435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '657fc5b1bd2b431f8a38d905bc964c2c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.673013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4e60ab63-079e-41b3-8b5c-50e6cac677eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '657fc5b1bd2b431f8a38d905bc964c2c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:20.678839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1060/2117] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:20 2022] DELETE /identity/v3/users/657fc5b1bd2b431f8a38d905bc964c2c => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:20.968250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:20.969812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:20.985053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1bbe88-8eca-4d28-8b37-d95b3b452ccf admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.046071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.046729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.047040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.047324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] PATH_INFO: `/v3/projects/e1cf25c107da47a5827d4e12f1625521` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.048570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1cf25c107da47a5827d4e12f1625521)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.050507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.104392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1cf25c107da47a5827d4e12f1625521', 'request_id': 'req-cb90959e-38b9-4788-9b85-6f2da15f720e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:21.113037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1cf25c107da47a5827d4e12f1625521', 'request_id': 'req-cb90959e-38b9-4788-9b85-6f2da15f720e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:21.119193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1cf25c107da47a5827d4e12f1625521', 'request_id': 'req-cb90959e-38b9-4788-9b85-6f2da15f720e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:21.131584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] The token cache is being invalidate because project e1cf25c107da47a5827d4e12f1625521 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:21.131867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:21.132519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cb90959e-38b9-4788-9b85-6f2da15f720e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:21.134061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1058/2118] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:20 2022] DELETE /identity/v3/projects/e1cf25c107da47a5827d4e12f1625521 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:21.139428 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610112d7-2c8f-4cd0-8a47-090e62b4b822 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.139904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610112d7-2c8f-4cd0-8a47-090e62b4b822 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.140013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610112d7-2c8f-4cd0-8a47-090e62b4b822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.160960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-610112d7-2c8f-4cd0-8a47-090e62b4b822 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:21.162653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-610112d7-2c8f-4cd0-8a47-090e62b4b822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:21.230745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1061/2119] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:21.238514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca62d4-f9d8-4b3f-bb8a-b5dc8466b50b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.238795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca62d4-f9d8-4b3f-bb8a-b5dc8466b50b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.239107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca62d4-f9d8-4b3f-bb8a-b5dc8466b50b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.259037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-09ca62d4-f9d8-4b3f-bb8a-b5dc8466b50b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:21.260306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-09ca62d4-f9d8-4b3f-bb8a-b5dc8466b50b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:21.277078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1059/2120] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:21.280185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610112d7-2c8f-4cd0-8a47-090e62b4b822 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.291677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610112d7-2c8f-4cd0-8a47-090e62b4b822 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.297073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.297846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.298129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.298401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.299140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:21.299476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.301800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.313957 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1062/2121] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:12:21.318359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ca62d4-f9d8-4b3f-bb8a-b5dc8466b50b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.328258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ca62d4-f9d8-4b3f-bb8a-b5dc8466b50b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.332898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.333505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.333782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.334051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.334713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.336671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.375000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1060/2122] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:21 2022] POST /identity/v3/projects => generated 385 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:21.376827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.386784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d1e729-eab5-40d9-a23f-e892767c648c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.425002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:21.445215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.445832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.446113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.446388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.447137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.449055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.470470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1063/2123] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:21 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:21.474659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.485396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3d7b49-92e8-4baf-b24e-873a8429c598 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.491086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.491712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.491995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.492263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.492957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.494701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.505361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1061/2124] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:21.509502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.520203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf4a18-2508-47d1-bdf3-0f617c79047d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.525355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8498e1a5-8884-4546-8003-17d5b99ddaff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.526006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8498e1a5-8884-4546-8003-17d5b99ddaff admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.526241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8498e1a5-8884-4546-8003-17d5b99ddaff admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.526546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8498e1a5-8884-4546-8003-17d5b99ddaff admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.527280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8498e1a5-8884-4546-8003-17d5b99ddaff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.528877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8498e1a5-8884-4546-8003-17d5b99ddaff admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.539178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1064/2125] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:12:21.543408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.553259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6801a8-5bf3-4036-a395-c27e71c00e41 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.558445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.559158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.559541 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.559819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] PATH_INFO: `/v3/projects/7b7cffb8483c4f0a82e361dbfdeaab89/users/c06a718f2ec94f98a7a650c450e247c8/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.575411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b7cffb8483c4f0a82e361dbfdeaab89, user_id=c06a718f2ec94f98a7a650c450e247c8, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.578449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.592323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1062/2126] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:21 2022] PUT /identity/v3/projects/7b7cffb8483c4f0a82e361dbfdeaab89/users/c06a718f2ec94f98a7a650c450e247c8/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:21.603839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2601c07-0ae1-405f-aa88-2b4a641a41ec None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.604132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2601c07-0ae1-405f-aa88-2b4a641a41ec None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.604387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2601c07-0ae1-405f-aa88-2b4a641a41ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.642719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b2601c07-0ae1-405f-aa88-2b4a641a41ec None None] MFA Rules not processed for user `c06a718f2ec94f98a7a650c450e247c8`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:21.642719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b2601c07-0ae1-405f-aa88-2b4a641a41ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:21.712258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1065/2127] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:21.716664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.721358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:21.794489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81762e05-9369-4e2f-b7db-3c394b2847be admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.801346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.802333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.802708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.803132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.804213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.806417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.837938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1063/2128] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:21 2022] POST /identity/v3/projects => generated 385 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:21.841892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2601c07-0ae1-405f-aa88-2b4a641a41ec None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.855149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2601c07-0ae1-405f-aa88-2b4a641a41ec None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.925295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.925934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.926215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.926631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.927405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.929392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:21.958828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1066/2129] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:21 2022] POST /identity/v3/users => generated 299 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:12:21.960311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:21.974750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58b47f9-28c5-4fea-aa0e-2d0d299029df admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:21.982325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:21.982979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:21.983282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:21.983562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:21.984261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:21.985905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:22.016549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1064/2130] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:22.021136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:22.041603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09746490-d717-40f9-be20-5d0b69b4ffba admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:22.048517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:22.049721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:22.050072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:22.050359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] PATH_INFO: `/v3/projects/92320cb7451f4c2d8ba0b382cca95615/users/5c21758333ce439ea8f284a77a448d50/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:22.069795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92320cb7451f4c2d8ba0b382cca95615, user_id=5c21758333ce439ea8f284a77a448d50, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:22.072182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:22.091225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1067/2131] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:22 2022] PUT /identity/v3/projects/92320cb7451f4c2d8ba0b382cca95615/users/5c21758333ce439ea8f284a77a448d50/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:22.095270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:22.115460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ba93c2-a841-4c8a-aafd-6adb3460f61e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:22.198638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f1b15b-224e-48fe-901c-ba4d6e019484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:22.199368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1b15b-224e-48fe-901c-ba4d6e019484 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:22.199666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1b15b-224e-48fe-901c-ba4d6e019484 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:22.199917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1b15b-224e-48fe-901c-ba4d6e019484 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:22.200837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f1b15b-224e-48fe-901c-ba4d6e019484 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:22.203135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f1b15b-224e-48fe-901c-ba4d6e019484 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:22.220497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1065/2132] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:22.222342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:22.234369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6a6950-561c-421f-a3ce-3b1ef4d1a934 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:22.239703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:22.240410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:22.240695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:22.240971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] PATH_INFO: `/v3/projects/92320cb7451f4c2d8ba0b382cca95615/users/5c21758333ce439ea8f284a77a448d50/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:22.243010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92320cb7451f4c2d8ba0b382cca95615, user_id=5c21758333ce439ea8f284a77a448d50, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:22.245749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:22.255027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:22.261985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1068/2133] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:22 2022] PUT /identity/v3/projects/92320cb7451f4c2d8ba0b382cca95615/users/5c21758333ce439ea8f284a77a448d50/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:12:22.272659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e174c41-e312-4491-a2ea-3e32df4e8bad None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:22.273104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e174c41-e312-4491-a2ea-3e32df4e8bad None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:22.273481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e174c41-e312-4491-a2ea-3e32df4e8bad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:22.312867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1e174c41-e312-4491-a2ea-3e32df4e8bad None None] MFA Rules not processed for user `5c21758333ce439ea8f284a77a448d50`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:22.314108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1e174c41-e312-4491-a2ea-3e32df4e8bad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:22.390076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:22.397382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1066/2134] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:22 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:22.412250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:22.415848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:22.496261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44204024-8404-44a5-ad41-afce6dcdd234 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:22.499855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43af5a9-f4ff-42d8-9109-b7533b12d2e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:22.500491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43af5a9-f4ff-42d8-9109-b7533b12d2e5 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:22.500765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43af5a9-f4ff-42d8-9109-b7533b12d2e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:22.501036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43af5a9-f4ff-42d8-9109-b7533b12d2e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:22.510162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43af5a9-f4ff-42d8-9109-b7533b12d2e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:22.511612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43af5a9-f4ff-42d8-9109-b7533b12d2e5 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:22.520817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1069/2135] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:22 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:23.133237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e174c41-e312-4491-a2ea-3e32df4e8bad None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:23.211309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e174c41-e312-4491-a2ea-3e32df4e8bad None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:23.216187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:23.216777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:23.217056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:23.217328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] PATH_INFO: `/v3/users/c06a718f2ec94f98a7a650c450e247c8` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:23.218488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c06a718f2ec94f98a7a650c450e247c8)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:23.220457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:23.272503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c06a718f2ec94f98a7a650c450e247c8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:23.281212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c06a718f2ec94f98a7a650c450e247c8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:23.290096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c06a718f2ec94f98a7a650c450e247c8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:23.303126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c06a718f2ec94f98a7a650c450e247c8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:23.303851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c06a718f2ec94f98a7a650c450e247c8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:23.310101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1067/2136] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:23 2022] DELETE /identity/v3/users/c06a718f2ec94f98a7a650c450e247c8 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:23.314685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab075e-42e5-439c-ae59-16e6c68756fe None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:23.315178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab075e-42e5-439c-ae59-16e6c68756fe None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:23.315284 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab075e-42e5-439c-ae59-16e6c68756fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:23.334217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-92ab075e-42e5-439c-ae59-16e6c68756fe None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:23.335441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-92ab075e-42e5-439c-ae59-16e6c68756fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:23.394624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1070/2137] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:23.404539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:23.406192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:23.467031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54161af9-f9af-4ce6-80a0-155257bfd127 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:23.471203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:23.471810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:23.472061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:23.472334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:23.483809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:23.485657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:23.497335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1068/2138] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:23.683444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:23.684379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:23.684379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:23.704731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:23.705815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:23.767564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1071/2139] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:12:23 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:23.813466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:23.815445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:23.838926 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:23.888316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:23.889883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:23.891838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba640a0-fa59-4529-9c08-3a7809d04b8d service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:23.898076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:23.898718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:23.899019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:23.899296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:23.911819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:23.914114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:23.915956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7684587-45b2-4c14-afd5-290a8ddb5868 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:23.921894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:23.922465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:23.922725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:23.923025 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] PATH_INFO: `/v3/projects/7b7cffb8483c4f0a82e361dbfdeaab89` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:23.924214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b7cffb8483c4f0a82e361dbfdeaab89)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:23.924406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1069/2140] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:12:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:12:23.932999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:23.984521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:23.989713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b7cffb8483c4f0a82e361dbfdeaab89', 'request_id': 'req-59942153-e5af-4b32-893f-29998fafe033'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.004003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b7cffb8483c4f0a82e361dbfdeaab89', 'request_id': 'req-59942153-e5af-4b32-893f-29998fafe033'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.011392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b7cffb8483c4f0a82e361dbfdeaab89', 'request_id': 'req-59942153-e5af-4b32-893f-29998fafe033'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.023160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] The token cache is being invalidate because project 7b7cffb8483c4f0a82e361dbfdeaab89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:24.023450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.024123 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-59942153-e5af-4b32-893f-29998fafe033 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.027241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1072/2141] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:23 2022] DELETE /identity/v3/projects/7b7cffb8483c4f0a82e361dbfdeaab89 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:12:24.029866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:24.042484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34797091-6c26-4f47-a056-0c25a0a87069 service swift] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:24.101529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:24.102187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.102473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.102753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] PATH_INFO: `/v3/users/5c21758333ce439ea8f284a77a448d50` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.104056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c21758333ce439ea8f284a77a448d50)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:24.106029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:24.158712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c21758333ce439ea8f284a77a448d50'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.166098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c21758333ce439ea8f284a77a448d50'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.173990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c21758333ce439ea8f284a77a448d50'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.187724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c21758333ce439ea8f284a77a448d50'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.188531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c21758333ce439ea8f284a77a448d50'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.196980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1070/2142] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:24 2022] DELETE /identity/v3/users/5c21758333ce439ea8f284a77a448d50 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:24.493997 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.494629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.494629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.501365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:24.503602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:24.516343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:24.517752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:24.537410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:24.578901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30fac12-baf7-4c95-b7d4-c82d3751599f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:24.584015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:24.584672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.585236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.585334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] PATH_INFO: `/v3/projects/92320cb7451f4c2d8ba0b382cca95615` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.586513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] RBAC: Authorizing `identity:delete_project(project_id=92320cb7451f4c2d8ba0b382cca95615)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:24.588588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:24.592854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1073/2143] 10.222.0.42 () {58 vars in 967 bytes} [Tue Jul 26 03:12:24 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:24.606028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:24.607551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:24.641117 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92320cb7451f4c2d8ba0b382cca95615', 'request_id': 'req-6d58eaee-6d94-42ef-af64-ddd082aa6154'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.648764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92320cb7451f4c2d8ba0b382cca95615', 'request_id': 'req-6d58eaee-6d94-42ef-af64-ddd082aa6154'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.655704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92320cb7451f4c2d8ba0b382cca95615', 'request_id': 'req-6d58eaee-6d94-42ef-af64-ddd082aa6154'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.668173 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] The token cache is being invalidate because project 92320cb7451f4c2d8ba0b382cca95615 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:24.668468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.669123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d58eaee-6d94-42ef-af64-ddd082aa6154 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:24.670710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1071/2144] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:24 2022] DELETE /identity/v3/projects/92320cb7451f4c2d8ba0b382cca95615 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:12:24.676148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41de01c-0810-4eb4-b379-1bd48a8de6aa None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.676496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41de01c-0810-4eb4-b379-1bd48a8de6aa None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.676892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41de01c-0810-4eb4-b379-1bd48a8de6aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.683985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2567dd-6a49-4cf3-a8c0-6e8162060109 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:24.698839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f41de01c-0810-4eb4-b379-1bd48a8de6aa None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:24.700104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f41de01c-0810-4eb4-b379-1bd48a8de6aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:24.771759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:24.772673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.773423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.773778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.776215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:24.794043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1072/2145] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:24.801899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a12fe44-e87c-44aa-8344-4b6c69a0e53e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.802181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a12fe44-e87c-44aa-8344-4b6c69a0e53e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.802459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a12fe44-e87c-44aa-8344-4b6c69a0e53e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.803299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:24.825681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1a12fe44-e87c-44aa-8344-4b6c69a0e53e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:24.827351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1a12fe44-e87c-44aa-8344-4b6c69a0e53e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:24.835831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1073/2146] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:24.840138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a12fe44-e87c-44aa-8344-4b6c69a0e53e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:24.848553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:24.850476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a12fe44-e87c-44aa-8344-4b6c69a0e53e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:24.850595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:24.855661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:24.856010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.856292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.856566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.857302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:24.857605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:24.859402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:24.861733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1074/2147] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:12:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 26 03:12:24.872246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1074/2148] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:12:24.876720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:24.888394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c94825-cef3-4924-81b9-11d0bb5eaa97 service swift] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:24.895317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:24.896019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:24.896425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:24.896708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:24.897399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:24.899383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:24.928113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1075/2149] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:24 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:24.932552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:24.943244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f470d2e2-7f45-4f67-a66b-584104be14ff admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:24.988745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:25.013925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.014490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.014776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.015089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.016222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.017737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.041592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1075/2150] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:24 2022] POST /identity/v3/users => generated 290 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:25.045970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.059495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d38d918-0fbf-4316-b841-66e73b698769 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.065841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.066619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.071371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.072010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.072496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.074579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.087946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1076/2151] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:25.093105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.104546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac02660-4c74-4855-8be2-48b2929ab536 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.110899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a137772-22ff-43c5-9f96-d04e24aaa5fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.111559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a137772-22ff-43c5-9f96-d04e24aaa5fb admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.111840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a137772-22ff-43c5-9f96-d04e24aaa5fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.112666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a137772-22ff-43c5-9f96-d04e24aaa5fb admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.113457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a137772-22ff-43c5-9f96-d04e24aaa5fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.115300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a137772-22ff-43c5-9f96-d04e24aaa5fb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.127072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1076/2152] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:25.131194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.142078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5eef2-3610-40a9-af47-346d201ca4fa admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.149336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a81dd57-c738-4565-9f55-9a6e4b2c2a32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.150103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a81dd57-c738-4565-9f55-9a6e4b2c2a32 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.150377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a81dd57-c738-4565-9f55-9a6e4b2c2a32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.150648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a81dd57-c738-4565-9f55-9a6e4b2c2a32 admin admin] PATH_INFO: `/v3/projects/ec59e88f21b248a7af0fae4782b8d512/users/924a536a4bfd42efb648bc265da5e873/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.154313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.154601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.154885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.166807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a81dd57-c738-4565-9f55-9a6e4b2c2a32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec59e88f21b248a7af0fae4782b8d512, user_id=924a536a4bfd42efb648bc265da5e873, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.169639 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a81dd57-c738-4565-9f55-9a6e4b2c2a32 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.178694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:25.180463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:25.185108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1077/2153] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:25 2022] PUT /identity/v3/projects/ec59e88f21b248a7af0fae4782b8d512/users/924a536a4bfd42efb648bc265da5e873/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:25.196070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27fcd0f-75f6-4fe7-a350-c34d50ad62e3 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.196380 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27fcd0f-75f6-4fe7-a350-c34d50ad62e3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.196631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27fcd0f-75f6-4fe7-a350-c34d50ad62e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.227838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a27fcd0f-75f6-4fe7-a350-c34d50ad62e3 None None] MFA Rules not processed for user `924a536a4bfd42efb648bc265da5e873`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:25.229950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a27fcd0f-75f6-4fe7-a350-c34d50ad62e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:25.245156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1077/2154] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:12:25 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:25.257043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.258398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:25.303818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1078/2155] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:25 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:25.308405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27fcd0f-75f6-4fe7-a350-c34d50ad62e3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.327545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eee01de-f384-459f-b16a-8a90bfe4b896 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.332573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.333202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.333478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.333756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.342416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.344055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.353094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1078/2156] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:12:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 26 03:12:25.359809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:25.372271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27fcd0f-75f6-4fe7-a350-c34d50ad62e3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.377487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.378200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.378549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.378911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.379631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.381824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.413790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1079/2157] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:25 2022] POST /identity/v3/projects => generated 381 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:12:25.415002 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.426361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dc3d37-44e5-46eb-96f7-e5a7bc9a1cc7 service swift] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.477376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:25.487569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.488197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.488500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.488756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.489438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.491530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.518653 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1079/2158] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:25 2022] POST /identity/v3/users => generated 297 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:25.520234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.532660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113a61c4-ccdb-43bb-81a4-4742e36621bb admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.538934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.539603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.539890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.540195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.540914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.542498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.553905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1080/2159] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:25.558359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.569665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c201b82-74f3-487d-8d2a-e7e38a82b7da admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.574873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.575617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.575901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.576178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] PATH_INFO: `/v3/projects/e8a0feb2138c4f5abe63816e6a0cc5e0/users/d0eca491b9394d44b330f7b1c6ab7c1d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.594845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8a0feb2138c4f5abe63816e6a0cc5e0, user_id=d0eca491b9394d44b330f7b1c6ab7c1d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.597643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.617979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1080/2160] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:25 2022] PUT /identity/v3/projects/e8a0feb2138c4f5abe63816e6a0cc5e0/users/d0eca491b9394d44b330f7b1c6ab7c1d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:25.619266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.631282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d40242-a4b6-4911-94d5-d9bf8af615b5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.698141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0214a9be-06a1-4804-8339-d58f90d56875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.698784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0214a9be-06a1-4804-8339-d58f90d56875 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.699143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0214a9be-06a1-4804-8339-d58f90d56875 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.699421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0214a9be-06a1-4804-8339-d58f90d56875 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.700128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0214a9be-06a1-4804-8339-d58f90d56875 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.701842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0214a9be-06a1-4804-8339-d58f90d56875 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.715517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1081/2161] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:25 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:12:25.716748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.723724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227cbd8-2392-43b8-85c3-4af0dc7574fc None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.724083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227cbd8-2392-43b8-85c3-4af0dc7574fc None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.724430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227cbd8-2392-43b8-85c3-4af0dc7574fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.727659 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f6ba1-8f43-4d87-bd0e-9520fc7f71d2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.732693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e694cae-0798-4608-bd68-beb8e25c6afb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.734284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e694cae-0798-4608-bd68-beb8e25c6afb admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.734631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e694cae-0798-4608-bd68-beb8e25c6afb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.734918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e694cae-0798-4608-bd68-beb8e25c6afb admin admin] PATH_INFO: `/v3/projects/e8a0feb2138c4f5abe63816e6a0cc5e0/users/d0eca491b9394d44b330f7b1c6ab7c1d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.737283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e694cae-0798-4608-bd68-beb8e25c6afb admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8a0feb2138c4f5abe63816e6a0cc5e0, user_id=d0eca491b9394d44b330f7b1c6ab7c1d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.740445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e694cae-0798-4608-bd68-beb8e25c6afb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.745842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-9227cbd8-2392-43b8-85c3-4af0dc7574fc None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:25.747447 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9227cbd8-2392-43b8-85c3-4af0dc7574fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:25.762244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1081/2162] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:25 2022] PUT /identity/v3/projects/e8a0feb2138c4f5abe63816e6a0cc5e0/users/d0eca491b9394d44b330f7b1c6ab7c1d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:25.764019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.764318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.764592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.797573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] MFA Rules not processed for user `d0eca491b9394d44b330f7b1c6ab7c1d`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:25.799043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:25.811284 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1082/2163] 10.222.0.42 () {58 vars in 967 bytes} [Tue Jul 26 03:12:25 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:25.824695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9227cbd8-2392-43b8-85c3-4af0dc7574fc None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.875191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1082/2164] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:25 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:25.886140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:25.887510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:25.897823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9227cbd8-2392-43b8-85c3-4af0dc7574fc None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.904593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.905437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.905875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.906166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.917724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.919829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.924112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:25.935663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1083/2165] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:12:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 26 03:12:25.952317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c88e9ee-99bf-45d3-bb01-be4a77a10ab7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:25.957897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cc85dc-2511-437f-8e99-ae28518bf527 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:25.958548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cc85dc-2511-437f-8e99-ae28518bf527 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:25.958829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cc85dc-2511-437f-8e99-ae28518bf527 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:25.959140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cc85dc-2511-437f-8e99-ae28518bf527 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:25.969445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cc85dc-2511-437f-8e99-ae28518bf527 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:25.971260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cc85dc-2511-437f-8e99-ae28518bf527 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:25.984221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1083/2166] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:25 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:26.405712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:26.474275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9900fccf-9af8-476b-b5bb-9f37288398d3 service swift] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:26.479115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:26.479786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:26.480068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:26.480343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] PATH_INFO: `/v3/users/924a536a4bfd42efb648bc265da5e873` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:26.481519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] RBAC: Authorizing `identity:delete_user(user_id=924a536a4bfd42efb648bc265da5e873)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:26.483516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:26.532915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '924a536a4bfd42efb648bc265da5e873'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.544581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '924a536a4bfd42efb648bc265da5e873'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.551333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '924a536a4bfd42efb648bc265da5e873'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.560159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '924a536a4bfd42efb648bc265da5e873'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.560793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '924a536a4bfd42efb648bc265da5e873'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.567159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1084/2167] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:26 2022] DELETE /identity/v3/users/924a536a4bfd42efb648bc265da5e873 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:26.572274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:26.572735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:26.572909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:26.593656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:26.594877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:26.628918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:26.669485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1084/2168] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:26.670900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:26.672347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:26.676153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:26.677534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:26.687212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb216f-ca44-4098-9494-43474e31c193 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:26.692616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:26.693395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:26.693758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:26.694112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] PATH_INFO: `/v3/users/d2e5f6a317494f14808b1036d50e1b0f` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:26.695443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2e5f6a317494f14808b1036d50e1b0f)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:26.697615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:26.753265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696ccc4e-d972-471a-8f4d-8df1603184db None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:26.757530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e695f5ef-c7ee-4c0c-958c-c47dbde3a02e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:26.757976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e695f5ef-c7ee-4c0c-958c-c47dbde3a02e service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:26.758261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e695f5ef-c7ee-4c0c-958c-c47dbde3a02e service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:26.758535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e695f5ef-c7ee-4c0c-958c-c47dbde3a02e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:26.764185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2e5f6a317494f14808b1036d50e1b0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.769107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e695f5ef-c7ee-4c0c-958c-c47dbde3a02e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:26.770579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e695f5ef-c7ee-4c0c-958c-c47dbde3a02e service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:26.782598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2e5f6a317494f14808b1036d50e1b0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.787787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1085/2169] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:26.788467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:26.791603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2e5f6a317494f14808b1036d50e1b0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.809419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2e5f6a317494f14808b1036d50e1b0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.810125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2e5f6a317494f14808b1036d50e1b0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:26.817691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1085/2170] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:26 2022] DELETE /identity/v3/users/d2e5f6a317494f14808b1036d50e1b0f => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:26.822491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:26.835020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:26.835377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:26.867474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:26.873268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:26.981592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1086/2171] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:27.003070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:27.007082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:27.095078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6157567b-e4f1-46bc-9826-e59cfd4970c4 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:27.098824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:27.099490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:27.099773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:27.100906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:27.115707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:27.115707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:27.117378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:27.119689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:27.129272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1086/2172] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:27.146242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f658c7a-fe6b-4964-b3f5-d76d74b78b90 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:27.151417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:27.152084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:27.152492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:27.152777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] PATH_INFO: `/v3/projects/ec59e88f21b248a7af0fae4782b8d512` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:27.153973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec59e88f21b248a7af0fae4782b8d512)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:27.155964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:27.212236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec59e88f21b248a7af0fae4782b8d512', 'request_id': 'req-aba2b45e-0e76-46fa-a681-64787948faac'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.221143 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec59e88f21b248a7af0fae4782b8d512', 'request_id': 'req-aba2b45e-0e76-46fa-a681-64787948faac'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.232897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec59e88f21b248a7af0fae4782b8d512', 'request_id': 'req-aba2b45e-0e76-46fa-a681-64787948faac'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.241372 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:27.247133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] The token cache is being invalidate because project ec59e88f21b248a7af0fae4782b8d512 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:27.247484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.249337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.251471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1087/2173] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:27 2022] DELETE /identity/v3/projects/ec59e88f21b248a7af0fae4782b8d512 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:27.255707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:27.266227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a217d19-64ab-41c8-91e5-b18e8c6f0c6e service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:27.354115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:27.363351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:27.363995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:27.364296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] PATH_INFO: `/v3/users/d0eca491b9394d44b330f7b1c6ab7c1d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:27.366706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d0eca491b9394d44b330f7b1c6ab7c1d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:27.369068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:27.436752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd0eca491b9394d44b330f7b1c6ab7c1d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.452389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd0eca491b9394d44b330f7b1c6ab7c1d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.461178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd0eca491b9394d44b330f7b1c6ab7c1d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.473906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd0eca491b9394d44b330f7b1c6ab7c1d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.475133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4949a0c6-3625-44ed-b6a6-cdb2cc91e4d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd0eca491b9394d44b330f7b1c6ab7c1d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.485389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1087/2174] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:27 2022] DELETE /identity/v3/users/d0eca491b9394d44b330f7b1c6ab7c1d => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:27.636334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:27.638188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:27.652304 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2b45e-0e76-46fa-a681-64787948faac admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:27.718938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:27.719655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:27.719914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:27.720190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] PATH_INFO: `/v3/projects/8327b64aa27b439bbacb285d1b4edbde` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:27.721336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8327b64aa27b439bbacb285d1b4edbde)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:27.723507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:27.770435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8327b64aa27b439bbacb285d1b4edbde', 'request_id': 'req-31bdad75-84c9-4e28-b1a5-f406d356cb23'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.779864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8327b64aa27b439bbacb285d1b4edbde', 'request_id': 'req-31bdad75-84c9-4e28-b1a5-f406d356cb23'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.786358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8327b64aa27b439bbacb285d1b4edbde', 'request_id': 'req-31bdad75-84c9-4e28-b1a5-f406d356cb23'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.799224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] The token cache is being invalidate because project 8327b64aa27b439bbacb285d1b4edbde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:27.799497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.800155 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-31bdad75-84c9-4e28-b1a5-f406d356cb23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:27.801692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1088/2175] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:27 2022] DELETE /identity/v3/projects/8327b64aa27b439bbacb285d1b4edbde => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:27.807301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247cb0b-cb20-46d8-a719-cbc0ce1c2613 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:27.807837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247cb0b-cb20-46d8-a719-cbc0ce1c2613 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:27.807837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247cb0b-cb20-46d8-a719-cbc0ce1c2613 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:27.829144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0247cb0b-cb20-46d8-a719-cbc0ce1c2613 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:27.830664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0247cb0b-cb20-46d8-a719-cbc0ce1c2613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:27.836594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:27.930673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1088/2176] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:27.938988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:27.938988 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:27.939272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:27.960959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:27.963156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:27.974399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1089/2177] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:27.979349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247cb0b-cb20-46d8-a719-cbc0ce1c2613 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:27.990059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247cb0b-cb20-46d8-a719-cbc0ce1c2613 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:27.990343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:27.997122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:27.997275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:27.997275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:27.997275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:27.997275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:27.997961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:27.998400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.000170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.014933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a653e8e-7f95-410c-94f8-74b10cbf1e81 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.022899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.023840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.024374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.024790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] PATH_INFO: `/v3/projects/e8a0feb2138c4f5abe63816e6a0cc5e0` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.025875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1089/2178] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:12:28.027260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8a0feb2138c4f5abe63816e6a0cc5e0)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.029858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.030502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.040638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adcd5b8-d219-4665-bd23-7ac11685e84c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.046507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.047488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.047841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.048121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.048831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.050832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.085205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1090/2179] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:12:28.085405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8a0feb2138c4f5abe63816e6a0cc5e0', 'request_id': 'req-8b3d1dc4-4306-40b8-b75a-997d80fb850b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.089568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.094455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8a0feb2138c4f5abe63816e6a0cc5e0', 'request_id': 'req-8b3d1dc4-4306-40b8-b75a-997d80fb850b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.101537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8a0feb2138c4f5abe63816e6a0cc5e0', 'request_id': 'req-8b3d1dc4-4306-40b8-b75a-997d80fb850b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.114496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] The token cache is being invalidate because project e8a0feb2138c4f5abe63816e6a0cc5e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:28.114922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.115967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8b3d1dc4-4306-40b8-b75a-997d80fb850b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.120936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1090/2180] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:27 2022] DELETE /identity/v3/projects/e8a0feb2138c4f5abe63816e6a0cc5e0 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:12:28.122758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.123046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.123373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.139585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:28.143978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:28.146135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.172877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98483a35-807c-4d92-8ee4-9fb0c0fe8262 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.180427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.181875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.182558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.183069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.184053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.186942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.212547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1091/2181] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:28.220253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1091/2182] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/users => generated 287 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:12:28.220253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.220429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.221493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.223813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.234916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa02a99a-4739-4097-8d25-0f3bcfbdffd6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.237265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb425b3-b1a7-443d-a1a1-2dd5cc988087 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.239950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.240782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.241106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.241479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] PATH_INFO: `/v3/users/f35bffdd694f4df7befe85d401191ff0` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.242775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f35bffdd694f4df7befe85d401191ff0)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.243092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f346873-405f-48a9-b236-62f3ccab28a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.243866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f346873-405f-48a9-b236-62f3ccab28a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.244151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f346873-405f-48a9-b236-62f3ccab28a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.244431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f346873-405f-48a9-b236-62f3ccab28a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.245145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f346873-405f-48a9-b236-62f3ccab28a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.245419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.246748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f346873-405f-48a9-b236-62f3ccab28a2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.259211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1092/2183] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:28 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:28.263703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbf983-8910-46b6-a810-1937c3efa75c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.264081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbf983-8910-46b6-a810-1937c3efa75c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.264454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbf983-8910-46b6-a810-1937c3efa75c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.287805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-99bbf983-8910-46b6-a810-1937c3efa75c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:28.289199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-99bbf983-8910-46b6-a810-1937c3efa75c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.298016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1093/2184] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:28.302090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f35bffdd694f4df7befe85d401191ff0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.302355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bbf983-8910-46b6-a810-1937c3efa75c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.312422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bbf983-8910-46b6-a810-1937c3efa75c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.316873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:28.317151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.318013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.318132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.318397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.319189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.319670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f35bffdd694f4df7befe85d401191ff0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.320996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.329775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f35bffdd694f4df7befe85d401191ff0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.332689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1094/2185] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:28 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:28.333511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.339818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f35bffdd694f4df7befe85d401191ff0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.340686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f35bffdd694f4df7befe85d401191ff0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.344434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d14dd48-34a2-4d83-b1a4-a82b56ae6708 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.347223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1092/2186] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:28 2022] DELETE /identity/v3/users/f35bffdd694f4df7befe85d401191ff0 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:28.347988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.349013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc289f6-c266-4180-9b39-61c361ef17c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.349369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.349621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc289f6-c266-4180-9b39-61c361ef17c1 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.349943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc289f6-c266-4180-9b39-61c361ef17c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.350169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc289f6-c266-4180-9b39-61c361ef17c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.350864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc289f6-c266-4180-9b39-61c361ef17c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:28.351205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc289f6-c266-4180-9b39-61c361ef17c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.352790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc289f6-c266-4180-9b39-61c361ef17c1 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.369612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1095/2187] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:12:28.369612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.369612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.369612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.387790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:28.389032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.415411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bae93d-e971-433f-b850-60ee0048ab7d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.419906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.420578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.420872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.421147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] PATH_INFO: `/v3/projects/9961303e55d24b0f96597e2bd4bda080/users/8586e2cdff7f48d08fdfee498c1f6c02/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.436963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9961303e55d24b0f96597e2bd4bda080, user_id=8586e2cdff7f48d08fdfee498c1f6c02, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.439676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.447107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1096/2188] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:28.447925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.449205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.454422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1093/2189] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:28 2022] PUT /identity/v3/projects/9961303e55d24b0f96597e2bd4bda080/users/8586e2cdff7f48d08fdfee498c1f6c02/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:28.458974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.462374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1fb66a-3780-43c0-9d5f-d2484e4e26e6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.465388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.521056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25514c63-4901-4ca7-984b-31b86a20b66e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.521693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25514c63-4901-4ca7-984b-31b86a20b66e admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.522088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25514c63-4901-4ca7-984b-31b86a20b66e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.522371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25514c63-4901-4ca7-984b-31b86a20b66e admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.523229 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25514c63-4901-4ca7-984b-31b86a20b66e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.524084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e7a326-3efd-4239-a833-2a0091624570 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.525243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25514c63-4901-4ca7-984b-31b86a20b66e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.528092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.528711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.528989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.529260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.537055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:28.543325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.545858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.555703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1097/2190] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:28.558219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1094/2191] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:28.559979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.560312 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08974151-6e60-4618-a958-43876236ccf8 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.560689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08974151-6e60-4618-a958-43876236ccf8 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.561059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08974151-6e60-4618-a958-43876236ccf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.571906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75d8647-1438-41f6-9ec3-3e3d85758816 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.590233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-08974151-6e60-4618-a958-43876236ccf8 None None] MFA Rules not processed for user `8586e2cdff7f48d08fdfee498c1f6c02`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:28.591670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-08974151-6e60-4618-a958-43876236ccf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.647627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.648253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.648531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.648803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.649481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.651469 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.661431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:28.667654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1098/2192] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:28.682316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08974151-6e60-4618-a958-43876236ccf8 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.683245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1095/2193] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/users => generated 289 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:12:28.684395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-08974151-6e60-4618-a958-43876236ccf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.685285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.695706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f35848-9c70-4371-9c0e-20b7e7032437 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.700961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.701579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.702212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.702803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.703371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.705146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.715813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1096/2194] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:12:28.720236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.731355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfb7faf-578e-47a2-ae86-e1b366af1432 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.736782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.737486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.737698 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.737979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.738720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.740543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.755596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1097/2195] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:28 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:28.756570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.758295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.759715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08974151-6e60-4618-a958-43876236ccf8 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.764175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.765126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.765415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.765672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.773571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f40fd7-3391-419f-ba48-1f9864bcb066 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.777812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.779335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.779950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.780963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.781314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.781505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] PATH_INFO: `/v3/projects/3c8d6492e3aa4191bf1554213bb749e9` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.782690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c8d6492e3aa4191bf1554213bb749e9)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.784813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.789189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1099/2196] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:28 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:28.790243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.802607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d53972-c161-44a9-a218-85a0b38e6437 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.807655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.808326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.812256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.812587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] PATH_INFO: `/v3/projects/4233aa1888f941fc8d56ceb4220d6950/users/1625ce5bebc046a6b59cc6b3c4da329e/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.830095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4233aa1888f941fc8d56ceb4220d6950, user_id=1625ce5bebc046a6b59cc6b3c4da329e, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.832905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.846874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c8d6492e3aa4191bf1554213bb749e9', 'request_id': 'req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.850476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1100/2197] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:28 2022] PUT /identity/v3/projects/4233aa1888f941fc8d56ceb4220d6950/users/1625ce5bebc046a6b59cc6b3c4da329e/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:28.851956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:28.861207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c8d6492e3aa4191bf1554213bb749e9', 'request_id': 'req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.867086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c8d6492e3aa4191bf1554213bb749e9', 'request_id': 'req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.878510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] The token cache is being invalidate because project 3c8d6492e3aa4191bf1554213bb749e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:28.878870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.879589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-67beaa88-a260-4b6b-aaf1-3f66a2a181ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:28.881137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1098/2198] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:28 2022] DELETE /identity/v3/projects/3c8d6492e3aa4191bf1554213bb749e9 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:28.882354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.882638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.882913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.919461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a65a7bd-6a20-4df6-97ac-48607c3549b7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:28.923637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] MFA Rules not processed for user `1625ce5bebc046a6b59cc6b3c4da329e`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:28.924584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:28.925036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:28.925349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:28.925634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:28.925915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] PATH_INFO: `/v3/users/8586e2cdff7f48d08fdfee498c1f6c02` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:28.927199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8586e2cdff7f48d08fdfee498c1f6c02)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:28.930475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:28.931667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:29.009442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8586e2cdff7f48d08fdfee498c1f6c02'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.020057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8586e2cdff7f48d08fdfee498c1f6c02'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.031560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8586e2cdff7f48d08fdfee498c1f6c02'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.039087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1099/2199] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:28 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:29.049528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:29.052199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:29.058091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8586e2cdff7f48d08fdfee498c1f6c02'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.059219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-49aa7e28-6797-4bce-a264-f302917d9537 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8586e2cdff7f48d08fdfee498c1f6c02'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.071157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1101/2200] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:28 2022] DELETE /identity/v3/users/8586e2cdff7f48d08fdfee498c1f6c02 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:29.074982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.075581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.077591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.109526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:29.110842 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:29.130937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:29.144608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c5b996-0c9f-4642-ba09-f7f93e2a61b5 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:29.178404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1102/2201] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:29.188768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:29.190243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:29.205964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79739bc-6b9f-4d2f-a135-c98d7e655458 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:29.206696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79739bc-6b9f-4d2f-a135-c98d7e655458 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.207008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79739bc-6b9f-4d2f-a135-c98d7e655458 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.207289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79739bc-6b9f-4d2f-a135-c98d7e655458 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.209188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e79739bc-6b9f-4d2f-a135-c98d7e655458 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:29.247284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab2f5b9-17fe-4b60-9dae-10c62f77f548 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:29.251360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:29.252247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.252364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.252555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.261784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:29.263265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:29.267874 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79739bc-6b9f-4d2f-a135-c98d7e655458 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:29.269312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79739bc-6b9f-4d2f-a135-c98d7e655458 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:29.272738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1103/2202] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:29.281038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1100/2203] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:29 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:12:29.560355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:29.561893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:29.581131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703bfdf4-a5c4-49b4-a08c-972e9ba1d701 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:29.590041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:29.590688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.591552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.591838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] PATH_INFO: `/v3/projects/9961303e55d24b0f96597e2bd4bda080` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.593487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9961303e55d24b0f96597e2bd4bda080)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:29.595462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:29.621999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:29.653276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9961303e55d24b0f96597e2bd4bda080', 'request_id': 'req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.661598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9961303e55d24b0f96597e2bd4bda080', 'request_id': 'req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.668031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9961303e55d24b0f96597e2bd4bda080', 'request_id': 'req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.679479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] The token cache is being invalidate because project 9961303e55d24b0f96597e2bd4bda080 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:29.679767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.680402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a38aeb69-57f9-4508-b43c-a2d1c4acd99a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:29.681992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1104/2204] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:29 2022] DELETE /identity/v3/projects/9961303e55d24b0f96597e2bd4bda080 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:29.686973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca1c33-8a33-4f90-8c34-480d5eb346b9 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.687264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca1c33-8a33-4f90-8c34-480d5eb346b9 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.687540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca1c33-8a33-4f90-8c34-480d5eb346b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.707312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-09ca1c33-8a33-4f90-8c34-480d5eb346b9 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:29.708434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-09ca1c33-8a33-4f90-8c34-480d5eb346b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:29.776693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1101/2205] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:29.784792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12aa13e-15d8-4466-8a49-1afc25601363 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.785071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12aa13e-15d8-4466-8a49-1afc25601363 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.785349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12aa13e-15d8-4466-8a49-1afc25601363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.807200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b12aa13e-15d8-4466-8a49-1afc25601363 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:29.808445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b12aa13e-15d8-4466-8a49-1afc25601363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:29.816923 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1105/2206] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:29.821480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ca1c33-8a33-4f90-8c34-480d5eb346b9 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:29.831932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ca1c33-8a33-4f90-8c34-480d5eb346b9 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:29.837362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:29.837988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.838272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.838553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.839300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:12:29.839608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:29.841236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:29.853154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1102/2207] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:12:29.857570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12aa13e-15d8-4466-8a49-1afc25601363 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:29.867364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12aa13e-15d8-4466-8a49-1afc25601363 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:29.872674 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:29.873295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.873600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.873874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.874573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:29.876566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:29.911416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1106/2208] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:29 2022] POST /identity/v3/projects => generated 371 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:29.915912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:29.926976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7e096-ffb9-44f7-99d5-f52945d30e4c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:29.968751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:29.987768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:29.988612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:29.989454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:29.989454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:29.989863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:29.991845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:30.014921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1103/2209] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:29 2022] POST /identity/v3/users => generated 286 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:30.019215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:30.030219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bbe4e1-c5d4-4776-af29-2dd642f696c7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:30.035903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:30.036513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:30.036788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:30.037050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:30.037746 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:30.039407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:30.049835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1107/2210] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:30.053954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:30.064901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20df0bdd-ed77-4efb-b7a7-9368aa7b9db7 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:30.070668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761aeafe-3688-4764-93bb-5c47f2a47a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:30.071304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761aeafe-3688-4764-93bb-5c47f2a47a29 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:30.071625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761aeafe-3688-4764-93bb-5c47f2a47a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:30.071924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761aeafe-3688-4764-93bb-5c47f2a47a29 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:30.072628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761aeafe-3688-4764-93bb-5c47f2a47a29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:30.074246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761aeafe-3688-4764-93bb-5c47f2a47a29 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:30.085179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1104/2211] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:30.091226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:30.105613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca708140-c42f-433d-b5f4-aa460c62df64 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:30.110995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:30.111672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:30.111952 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:30.112237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] PATH_INFO: `/v3/projects/bd8a79690708464a9c3ffe381667a03c/users/8697f1ada4214e15876b2c2ec6184ed7/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:30.129721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd8a79690708464a9c3ffe381667a03c, user_id=8697f1ada4214e15876b2c2ec6184ed7, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:30.132711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:30.148057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1108/2212] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:30 2022] PUT /identity/v3/projects/bd8a79690708464a9c3ffe381667a03c/users/8697f1ada4214e15876b2c2ec6184ed7/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:30.153830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cf749d-4853-418d-8a62-30d4739a0bca None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:30.154098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cf749d-4853-418d-8a62-30d4739a0bca None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:30.154406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cf749d-4853-418d-8a62-30d4739a0bca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:30.174311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-47cf749d-4853-418d-8a62-30d4739a0bca None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:30.175649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-47cf749d-4853-418d-8a62-30d4739a0bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:30.237654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1105/2213] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:30.248166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:30.250020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:30.307155 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11c1ace-b8db-4d40-8aef-8348766957d2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:30.310990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:30.311612 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:30.311893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:30.312167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:30.321230 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:30.322825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:30.332599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1109/2214] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:34.461400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcad25f6-d4c7-4902-b1a4-1f66f2cbd0c8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:34.461967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcad25f6-d4c7-4902-b1a4-1f66f2cbd0c8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:34.461967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcad25f6-d4c7-4902-b1a4-1f66f2cbd0c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:34.490892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-bcad25f6-d4c7-4902-b1a4-1f66f2cbd0c8 None None] MFA Rules not processed for user `8697f1ada4214e15876b2c2ec6184ed7`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:34.492468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bcad25f6-d4c7-4902-b1a4-1f66f2cbd0c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:34.568010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1106/2215] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:34 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:34.574064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:34.592053 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d5cd92-4313-4469-aced-6102bbaf4ab5 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:34.596569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:34.597181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:34.597456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:34.597812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:34.598415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:34.600411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:34.634351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1110/2216] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:12:34 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:34.638453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcad25f6-d4c7-4902-b1a4-1f66f2cbd0c8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:34.650177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcad25f6-d4c7-4902-b1a4-1f66f2cbd0c8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:34.706027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:34.718195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:34.718836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:34.719156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:34.719437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:34.720147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:34.722099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:34.749563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1107/2217] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:12:34 2022] POST /identity/v3/users => generated 293 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:12:34.750826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:34.762786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233c91dd-7707-4697-a434-6676e7a838b3 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:34.767759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:34.768592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:34.769308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:34.769762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:34.770675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:34.772807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:34.784949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1111/2218] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:34 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:34.789298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:34.802476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb81ad6-b8ec-4519-ac43-c02c2507b547 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:34.807956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:34.808598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:34.808886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:34.809172 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] PATH_INFO: `/v3/projects/19282dfa2407442c95f4abc8b5a7e6a4/users/82fb3f03efb64adc84081706eb92eb9e/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:34.826233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19282dfa2407442c95f4abc8b5a7e6a4, user_id=82fb3f03efb64adc84081706eb92eb9e, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:34.830293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:34.847633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1108/2219] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:34 2022] PUT /identity/v3/projects/19282dfa2407442c95f4abc8b5a7e6a4/users/82fb3f03efb64adc84081706eb92eb9e/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:34.852318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:34.862852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d212763-47d9-4b58-a533-ab754bd6f517 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:34.872426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:34.926915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db7a773-c13a-4473-8357-b0b9affe5b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:34.927719 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db7a773-c13a-4473-8357-b0b9affe5b82 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:34.927998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db7a773-c13a-4473-8357-b0b9affe5b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:34.928271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db7a773-c13a-4473-8357-b0b9affe5b82 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:34.928978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db7a773-c13a-4473-8357-b0b9affe5b82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:34.930632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db7a773-c13a-4473-8357-b0b9affe5b82 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:34.948302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1112/2220] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:12:34 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:12:34.953400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:34.965928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c306847-cb5a-45e9-bddd-74dc1392bbb9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:34.974493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24690526-f021-432b-a469-fe848934daf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:34.975235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24690526-f021-432b-a469-fe848934daf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:34.975501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24690526-f021-432b-a469-fe848934daf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:34.975773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24690526-f021-432b-a469-fe848934daf1 admin admin] PATH_INFO: `/v3/projects/19282dfa2407442c95f4abc8b5a7e6a4/users/82fb3f03efb64adc84081706eb92eb9e/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:34.977861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24690526-f021-432b-a469-fe848934daf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19282dfa2407442c95f4abc8b5a7e6a4, user_id=82fb3f03efb64adc84081706eb92eb9e, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:34.984774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24690526-f021-432b-a469-fe848934daf1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:35.000212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1109/2221] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:12:34 2022] PUT /identity/v3/projects/19282dfa2407442c95f4abc8b5a7e6a4/users/82fb3f03efb64adc84081706eb92eb9e/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:37.091201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:37.092114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:37.092626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:37.116130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] MFA Rules not processed for user `1a1125811ba549e0ae66b26ad3949f23`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:37.117589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:37.192349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1113/2222] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:37 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:42.343711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c69adf2-1ef1-4514-bae0-99180b397892 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:42.344724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c69adf2-1ef1-4514-bae0-99180b397892 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:42.345430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c69adf2-1ef1-4514-bae0-99180b397892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:42.425352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1c69adf2-1ef1-4514-bae0-99180b397892 None None] MFA Rules not processed for user `82fb3f03efb64adc84081706eb92eb9e`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:42.427472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1c69adf2-1ef1-4514-bae0-99180b397892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:42.621017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1110/2223] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:42 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:42.633321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:42.637537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:42.770796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dafc8e4-9230-44c8-b208-75c64d3e46a9 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:42.777747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdf156-1623-4f7f-8627-037492bdf954 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:42.778362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdf156-1623-4f7f-8627-037492bdf954 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:42.778642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdf156-1623-4f7f-8627-037492bdf954 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:42.778920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdf156-1623-4f7f-8627-037492bdf954 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:42.859222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:42.872428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdf156-1623-4f7f-8627-037492bdf954 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:42.875943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdf156-1623-4f7f-8627-037492bdf954 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:42.888537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1114/2224] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:44.469212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:44.470234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:44.470600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:44.491448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] MFA Rules not processed for user `8697f1ada4214e15876b2c2ec6184ed7`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:44.492666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:44.511652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1111/2225] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:44 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:50.679792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c1e236-9d3b-4d6d-85da-9fa69c244373 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:50.680334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c1e236-9d3b-4d6d-85da-9fa69c244373 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:50.680436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c1e236-9d3b-4d6d-85da-9fa69c244373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:50.704011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-84c1e236-9d3b-4d6d-85da-9fa69c244373 None None] MFA Rules not processed for user `9d3fd2b3c6cd47019e15fbf959d47381`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:50.705353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-84c1e236-9d3b-4d6d-85da-9fa69c244373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:50.762933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1115/2226] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:51.227371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:51.230251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:51.353058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26a1641-db73-47c5-aeef-2f6ade5de70d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:51.370296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:51.371077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:51.371426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:51.371775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] PATH_INFO: `/v3/users/1625ce5bebc046a6b59cc6b3c4da329e` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:51.373066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1625ce5bebc046a6b59cc6b3c4da329e)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:51.375202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:51.411791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:51.454103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1625ce5bebc046a6b59cc6b3c4da329e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:51.472888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1625ce5bebc046a6b59cc6b3c4da329e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:51.487651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1625ce5bebc046a6b59cc6b3c4da329e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:51.496937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:51.497217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:51.497624 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:51.502475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1625ce5bebc046a6b59cc6b3c4da329e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:51.503347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-407674e9-6cbb-418a-b7ec-37dc9579ca93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1625ce5bebc046a6b59cc6b3c4da329e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:51.517411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:51.518004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1112/2227] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:12:51 2022] DELETE /identity/v3/users/1625ce5bebc046a6b59cc6b3c4da329e => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:12:51.519235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:51.524683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:51.525060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:51.525473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:51.546240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:12:51.547526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:51.588226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1116/2228] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:12:51 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:12:51.601502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:51.603209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:51.649396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1113/2229] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:12:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:12:51.668242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:51.670293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:51.684362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ea91b2-8dcd-4e10-9b91-485ce4ad4498 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:51.695720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:51.697034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:51.697329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:51.697634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:51.720451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:51.722186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:51.743965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1117/2230] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:12:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 26 03:12:51.761765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14c8fde-cf8f-4d6c-bda1-091134ae2f6f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:51.773271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114b4386-8293-49dd-8396-19a623eacdf7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:51.775228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114b4386-8293-49dd-8396-19a623eacdf7 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:51.775911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114b4386-8293-49dd-8396-19a623eacdf7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:51.776375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114b4386-8293-49dd-8396-19a623eacdf7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:51.802347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114b4386-8293-49dd-8396-19a623eacdf7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:51.804339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114b4386-8293-49dd-8396-19a623eacdf7 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:51.817746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1114/2231] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:12:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:12:52.340985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:12:52.343012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:12:52.399476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d714af-6b3a-42da-8ab0-11a8739d5837 service swift] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:12:52.406370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:12:52.407521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:12:52.408035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:12:52.408516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] PATH_INFO: `/v3/projects/4233aa1888f941fc8d56ceb4220d6950` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:12:52.411161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] RBAC: Authorizing `identity:delete_project(project_id=4233aa1888f941fc8d56ceb4220d6950)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:12:52.413854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:12:52.452205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:12:52.489239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4233aa1888f941fc8d56ceb4220d6950', 'request_id': 'req-35cce4ec-1b2d-431c-9f83-8c204176427e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:52.515302 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4233aa1888f941fc8d56ceb4220d6950', 'request_id': 'req-35cce4ec-1b2d-431c-9f83-8c204176427e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:52.529592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4233aa1888f941fc8d56ceb4220d6950', 'request_id': 'req-35cce4ec-1b2d-431c-9f83-8c204176427e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:52.551717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] The token cache is being invalidate because project 4233aa1888f941fc8d56ceb4220d6950 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:12:52.552176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:52.553868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-35cce4ec-1b2d-431c-9f83-8c204176427e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:12:52.557515 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1118/2232] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:12:52 2022] DELETE /identity/v3/projects/4233aa1888f941fc8d56ceb4220d6950 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:13:01.813335 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=73,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:13:02.544694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=15,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:13:09.934326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292aa344-b2cc-4590-b44f-25adfbe50445 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:09.935016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292aa344-b2cc-4590-b44f-25adfbe50445 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:09.935720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292aa344-b2cc-4590-b44f-25adfbe50445 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:09.937493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1115/2233] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:13:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:13:09.943135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4b2e7-ab31-4fbf-b872-a543c77eadb1 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:09.944102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4b2e7-ab31-4fbf-b872-a543c77eadb1 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:09.944553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4b2e7-ab31-4fbf-b872-a543c77eadb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:09.969174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-59a4b2e7-ab31-4fbf-b872-a543c77eadb1 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:13:09.971461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-59a4b2e7-ab31-4fbf-b872-a543c77eadb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:10.065303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1119/2234] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:13:09 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:13:10.080793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292aa344-b2cc-4590-b44f-25adfbe50445 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:13:10.082638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-292aa344-b2cc-4590-b44f-25adfbe50445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:10.184215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292aa344-b2cc-4590-b44f-25adfbe50445 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:13:10.187863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0794af1b-c2b0-41c6-bdab-b98ddb555de8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:13:10.188486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0794af1b-c2b0-41c6-bdab-b98ddb555de8 service glance] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:10.188760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0794af1b-c2b0-41c6-bdab-b98ddb555de8 service glance] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:10.189035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0794af1b-c2b0-41c6-bdab-b98ddb555de8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:10.202996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0794af1b-c2b0-41c6-bdab-b98ddb555de8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:13:10.204596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0794af1b-c2b0-41c6-bdab-b98ddb555de8 service glance] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:13:10.216748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1116/2235] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:13:10 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:13:10.298415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996656a4-17f5-4dd3-801b-217088d18f64 None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:10.298717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996656a4-17f5-4dd3-801b-217088d18f64 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:10.299016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996656a4-17f5-4dd3-801b-217088d18f64 None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:10.300554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1120/2236] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:13:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:13:10.306228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c568ca-fc6d-4809-81e7-f5dcd008c39d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:10.306629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c568ca-fc6d-4809-81e7-f5dcd008c39d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:10.306993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c568ca-fc6d-4809-81e7-f5dcd008c39d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:10.327516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a3c568ca-fc6d-4809-81e7-f5dcd008c39d None None] MFA Rules not processed for user `29a1868eb95f4396aa3fd2da677b40f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:13:10.328754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a3c568ca-fc6d-4809-81e7-f5dcd008c39d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:10.381637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1117/2237] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:13:10 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:13:10.393538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996656a4-17f5-4dd3-801b-217088d18f64 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:13:10.395479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-996656a4-17f5-4dd3-801b-217088d18f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:10.409440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996656a4-17f5-4dd3-801b-217088d18f64 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:13:10.413045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572be3a1-747d-4a0b-b7c8-a55e2ad27495 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:13:10.413727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572be3a1-747d-4a0b-b7c8-a55e2ad27495 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:10.414013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572be3a1-747d-4a0b-b7c8-a55e2ad27495 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:10.414288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572be3a1-747d-4a0b-b7c8-a55e2ad27495 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:10.422089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572be3a1-747d-4a0b-b7c8-a55e2ad27495 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:13:10.423614 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572be3a1-747d-4a0b-b7c8-a55e2ad27495 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:13:10.432260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1121/2238] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:13:10 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:13:10.514825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7b26e0-b8aa-4e4c-ba70-70066319bb4e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:10.515151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7b26e0-b8aa-4e4c-ba70-70066319bb4e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:10.515431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7b26e0-b8aa-4e4c-ba70-70066319bb4e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:10.516887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1118/2239] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:13:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:13:10.522535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826990-190e-46d3-8dfb-5f52e156e18c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:10.522865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826990-190e-46d3-8dfb-5f52e156e18c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:10.523114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826990-190e-46d3-8dfb-5f52e156e18c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:10.543742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-55826990-190e-46d3-8dfb-5f52e156e18c None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:13:10.545093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-55826990-190e-46d3-8dfb-5f52e156e18c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:10.554620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1122/2240] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:13:10 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:13:10.565471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7b26e0-b8aa-4e4c-ba70-70066319bb4e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:13:10.567439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4e7b26e0-b8aa-4e4c-ba70-70066319bb4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:10.582683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7b26e0-b8aa-4e4c-ba70-70066319bb4e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:13:10.587126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:13:10.587751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:10.588029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:10.588307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:10.598673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:13:10.600186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:13:10.611359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1119/2241] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:13:10 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:13:13.210592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b43bfb3-ede7-4dcd-a421-9919e4c181e9 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:13.211030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b43bfb3-ede7-4dcd-a421-9919e4c181e9 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:13.211192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b43bfb3-ede7-4dcd-a421-9919e4c181e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:13.231681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1b43bfb3-ede7-4dcd-a421-9919e4c181e9 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:13:13.232806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1b43bfb3-ede7-4dcd-a421-9919e4c181e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:13.289084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1123/2242] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:13:13 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:13:13.301128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:13:13.302555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:13.370984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c286b4d-cef0-4170-80ce-cc87724fff79 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:13:13.377161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc2d773-d534-40dd-b4d4-073a9fb2fd2a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:13:13.377786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc2d773-d534-40dd-b4d4-073a9fb2fd2a service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:13.378069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc2d773-d534-40dd-b4d4-073a9fb2fd2a service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:13.378353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc2d773-d534-40dd-b4d4-073a9fb2fd2a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:13.387969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc2d773-d534-40dd-b4d4-073a9fb2fd2a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:13:13.389545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc2d773-d534-40dd-b4d4-073a9fb2fd2a service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:13:13.399879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1120/2243] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:13:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:13:23.281765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=65 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:13:23.395338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:13:32.143771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b43bfb3-ede7-4dcd-a421-9919e4c181e9 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:13:32.153691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b43bfb3-ede7-4dcd-a421-9919e4c181e9 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:13:32.157813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:13:32.158426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:32.158700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:32.158974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:32.160907 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:32.229226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:13:32.230673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:13:32.240765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1124/2244] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:13:32 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:13:37.434884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:37.435426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:37.435426 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:37.457620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:13:37.458851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:37.467493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1121/2245] 10.222.0.42 () {58 vars in 967 bytes} [Tue Jul 26 03:13:37 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:13:37.490828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:13:37.502563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb30802-15fc-479a-a89a-fa49ab84f644 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:13:37.507895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcc6e4f-8db0-4013-8af0-c9fd30be9cf2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:13:37.508524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc6e4f-8db0-4013-8af0-c9fd30be9cf2 service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:37.508805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc6e4f-8db0-4013-8af0-c9fd30be9cf2 service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:37.509084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc6e4f-8db0-4013-8af0-c9fd30be9cf2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:37.517992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcc6e4f-8db0-4013-8af0-c9fd30be9cf2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:13:37.519661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcc6e4f-8db0-4013-8af0-c9fd30be9cf2 service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:13:37.527999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1125/2246] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:13:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:13:42.789162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:13:42.790642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:42.863061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a32d42e-58f2-45ec-993c-5b7dc9cb832b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:13:42.868381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca606bc-d150-4533-a468-01158ded98cc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:13:42.869003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca606bc-d150-4533-a468-01158ded98cc service placement] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:13:42.869282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca606bc-d150-4533-a468-01158ded98cc service placement] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:13:42.869553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca606bc-d150-4533-a468-01158ded98cc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:13:42.871529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fca606bc-d150-4533-a468-01158ded98cc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:13:42.882915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca606bc-d150-4533-a468-01158ded98cc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:13:42.884527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca606bc-d150-4533-a468-01158ded98cc service placement] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:13:42.894516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1122/2247] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:13:42 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:13:47.524924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=36 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:13:52.890234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=35 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:14:05.792998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56e3dd-ab42-485d-b2d2-36131acae580 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:14:05.793536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56e3dd-ab42-485d-b2d2-36131acae580 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:14:05.793630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56e3dd-ab42-485d-b2d2-36131acae580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:14:05.813222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-fd56e3dd-ab42-485d-b2d2-36131acae580 None None] MFA Rules not processed for user `9d3fd2b3c6cd47019e15fbf959d47381`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:14:05.814420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fd56e3dd-ab42-485d-b2d2-36131acae580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:14:05.874200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1126/2248] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:14:05 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:14:09.069830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:14:09.070378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:14:09.070713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:14:09.092119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] MFA Rules not processed for user `1a1125811ba549e0ae66b26ad3949f23`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:14:09.093320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:14:09.150396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1123/2249] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:14:09 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:14:15.866326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=25 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:14:19.143340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=25 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:14:28.571921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd16501-89d9-4566-bbcf-8f52e0715938 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:14:28.572772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd16501-89d9-4566-bbcf-8f52e0715938 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:14:28.572772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd16501-89d9-4566-bbcf-8f52e0715938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:14:28.594291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3fd16501-89d9-4566-bbcf-8f52e0715938 None None] MFA Rules not processed for user `037c9f8229a44de198ebc3752f7a1966`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:14:28.595548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3fd16501-89d9-4566-bbcf-8f52e0715938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:14:28.653479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1127/2250] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:14:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:14:38.645677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=25 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:16.782228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:16.783518 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:16.873025 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c04148-c6a8-4fa1-8d4f-d81f76cb835b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:16.877832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:16.878679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:16.878981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:16.879440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] PATH_INFO: `/v3/users/1a1125811ba549e0ae66b26ad3949f23` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:16.880664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a1125811ba549e0ae66b26ad3949f23)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:16.882669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:16.931638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a1125811ba549e0ae66b26ad3949f23'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:16.943872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a1125811ba549e0ae66b26ad3949f23'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:16.950896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a1125811ba549e0ae66b26ad3949f23'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:16.958516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a1125811ba549e0ae66b26ad3949f23'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:16.959199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7ecc8b2b-40fb-4d5c-9110-0dcc28b41a57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a1125811ba549e0ae66b26ad3949f23'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:16.965131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1124/2251] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:15:16 2022] DELETE /identity/v3/users/1a1125811ba549e0ae66b26ad3949f23 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:17.182693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd16501-89d9-4566-bbcf-8f52e0715938 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:17.196340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd16501-89d9-4566-bbcf-8f52e0715938 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:17.265050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:17.265637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.265927 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.266224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] PATH_INFO: `/v3/projects/a07cd1888316403eb9c37610a310b76d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.267690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a07cd1888316403eb9c37610a310b76d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:17.269703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:17.329179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a07cd1888316403eb9c37610a310b76d', 'request_id': 'req-c875aa43-1c52-466b-968a-8b091b46854b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:17.338146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a07cd1888316403eb9c37610a310b76d', 'request_id': 'req-c875aa43-1c52-466b-968a-8b091b46854b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:17.344442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a07cd1888316403eb9c37610a310b76d', 'request_id': 'req-c875aa43-1c52-466b-968a-8b091b46854b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:17.358845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] The token cache is being invalidate because project a07cd1888316403eb9c37610a310b76d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:15:17.359279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:17.360020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c875aa43-1c52-466b-968a-8b091b46854b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:17.363504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1128/2252] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:15:17 2022] DELETE /identity/v3/projects/a07cd1888316403eb9c37610a310b76d => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:17.368598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91514940-5aa4-4cdb-8905-c01b4d9aa564 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.368746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91514940-5aa4-4cdb-8905-c01b4d9aa564 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.369058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91514940-5aa4-4cdb-8905-c01b4d9aa564 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.388958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-91514940-5aa4-4cdb-8905-c01b4d9aa564 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:17.390111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-91514940-5aa4-4cdb-8905-c01b4d9aa564 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:17.467860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1125/2253] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:15:17.476953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd95e0-d984-49b8-962e-5efaca09ca50 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.477242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd95e0-d984-49b8-962e-5efaca09ca50 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.477523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd95e0-d984-49b8-962e-5efaca09ca50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.497901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ccbd95e0-d984-49b8-962e-5efaca09ca50 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:17.499162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ccbd95e0-d984-49b8-962e-5efaca09ca50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:17.508467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1129/2254] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:17.512925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91514940-5aa4-4cdb-8905-c01b4d9aa564 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:17.523118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91514940-5aa4-4cdb-8905-c01b4d9aa564 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:17.528022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:17.528663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.528932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.529198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.529902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:15:17.530201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:17.531837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:17.543924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1126/2255] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:15:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:15:17.548599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbd95e0-d984-49b8-962e-5efaca09ca50 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:17.558194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbd95e0-d984-49b8-962e-5efaca09ca50 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:17.563683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:17.564580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.564826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.565148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.565811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:17.567802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:17.599177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1130/2256] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:15:17 2022] POST /identity/v3/projects => generated 367 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:17.603610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:17.614393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9da43fb-4e32-4d0e-a688-787f8f147924 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:17.618500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:17.677721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:17.678393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.678664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.678938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.679647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:17.681598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:17.712557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1127/2257] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:15:17 2022] POST /identity/v3/users => generated 284 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:17.714231 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:17.732876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7381676c-5990-410b-91ce-a0147b9f8919 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:17.737773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:17.738396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.738676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.738941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.739682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:17.741462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:17.755114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1131/2258] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:17.756622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:17.767695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20de8d86-8d7e-4aa3-9fda-80f2d82c9a55 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:17.774014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9bdfa3-a1eb-4cab-8992-5323e5137212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:17.774580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bdfa3-a1eb-4cab-8992-5323e5137212 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.774863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bdfa3-a1eb-4cab-8992-5323e5137212 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.775177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bdfa3-a1eb-4cab-8992-5323e5137212 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.775885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9bdfa3-a1eb-4cab-8992-5323e5137212 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:17.777513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9bdfa3-a1eb-4cab-8992-5323e5137212 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:17.789317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1128/2259] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:17 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:17.794580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:17.805481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174e168b-d224-4915-9771-99bcf36b71c2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:17.811061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:17.811815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.812124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.812391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] PATH_INFO: `/v3/projects/8e6cfe29946d49bfa9afcdb9dc3aa8fc/users/b47a7296baf14d798a50f837d2625f3e/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.829896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e6cfe29946d49bfa9afcdb9dc3aa8fc, user_id=b47a7296baf14d798a50f837d2625f3e, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:17.832744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:17.846552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1132/2260] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:17 2022] PUT /identity/v3/projects/8e6cfe29946d49bfa9afcdb9dc3aa8fc/users/b47a7296baf14d798a50f837d2625f3e/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:17.857496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5348e2-e83f-4efa-8ef5-874ca6721c3a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:17.857777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5348e2-e83f-4efa-8ef5-874ca6721c3a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:17.858083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5348e2-e83f-4efa-8ef5-874ca6721c3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:17.891391 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5d5348e2-e83f-4efa-8ef5-874ca6721c3a None None] MFA Rules not processed for user `b47a7296baf14d798a50f837d2625f3e`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:17.892620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5d5348e2-e83f-4efa-8ef5-874ca6721c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:17.985211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1129/2261] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:17 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:18.006913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:18.011844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:18.047120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:18.082165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5147a3-05a5-4773-8938-8635f8d9a020 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:18.089627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35650ab9-940b-4776-b77a-ccb331fe37b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:18.089627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35650ab9-940b-4776-b77a-ccb331fe37b0 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:18.089934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35650ab9-940b-4776-b77a-ccb331fe37b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:18.090153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35650ab9-940b-4776-b77a-ccb331fe37b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:18.104311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35650ab9-940b-4776-b77a-ccb331fe37b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:18.107050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35650ab9-940b-4776-b77a-ccb331fe37b0 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:18.121311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1133/2262] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:18 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:22.104452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c173155-5637-4af3-b2a2-4e030feb1e6d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:22.105013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c173155-5637-4af3-b2a2-4e030feb1e6d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:22.105487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c173155-5637-4af3-b2a2-4e030feb1e6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:22.125088 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0c173155-5637-4af3-b2a2-4e030feb1e6d None None] MFA Rules not processed for user `fbdac72f58df4335b3790c5fb1609f5b`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:22.126474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0c173155-5637-4af3-b2a2-4e030feb1e6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:22.183526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1130/2263] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:22 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:22.469536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4161a8-828e-4c8d-9fd6-ea6981759c3b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:22.470111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4161a8-828e-4c8d-9fd6-ea6981759c3b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:22.470111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4161a8-828e-4c8d-9fd6-ea6981759c3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:22.490030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6c4161a8-828e-4c8d-9fd6-ea6981759c3b None None] MFA Rules not processed for user `9d3fd2b3c6cd47019e15fbf959d47381`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:22.491331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6c4161a8-828e-4c8d-9fd6-ea6981759c3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:22.555731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1134/2264] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:22 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:27.253461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c173155-5637-4af3-b2a2-4e030feb1e6d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:27.274846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:27.317317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c173155-5637-4af3-b2a2-4e030feb1e6d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:27.321953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:27.322619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:27.322899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:27.323204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] PATH_INFO: `/v3/users/b47a7296baf14d798a50f837d2625f3e` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:27.324941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b47a7296baf14d798a50f837d2625f3e)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:27.326895 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:27.386046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b47a7296baf14d798a50f837d2625f3e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:27.401569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b47a7296baf14d798a50f837d2625f3e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:27.410522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b47a7296baf14d798a50f837d2625f3e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:27.422305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b47a7296baf14d798a50f837d2625f3e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:27.423233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b47a7296baf14d798a50f837d2625f3e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:27.434457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1131/2265] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:15:27 2022] DELETE /identity/v3/users/b47a7296baf14d798a50f837d2625f3e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:27.436387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:27.436872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:27.437227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:27.458942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:27.460321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:27.535168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1135/2266] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:27.560358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:27.562127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:27.639591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed08793-1923-46f8-af5f-5fe6ed4a337a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:27.643741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fdcc8f-adf8-4290-a0e0-6a191c5198ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:27.644371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fdcc8f-adf8-4290-a0e0-6a191c5198ce service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:27.644648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fdcc8f-adf8-4290-a0e0-6a191c5198ce service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:27.644921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fdcc8f-adf8-4290-a0e0-6a191c5198ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:27.655087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fdcc8f-adf8-4290-a0e0-6a191c5198ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:27.656542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fdcc8f-adf8-4290-a0e0-6a191c5198ce service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:27.666655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1132/2267] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:15:27.944618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:27.946514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:27.973318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e68b8d-aa46-45ae-8380-c2aafdc6ba99 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:27.978999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:27.979746 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:27.979939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:27.980217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] PATH_INFO: `/v3/projects/8e6cfe29946d49bfa9afcdb9dc3aa8fc` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:27.981393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e6cfe29946d49bfa9afcdb9dc3aa8fc)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:27.983360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.060553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e6cfe29946d49bfa9afcdb9dc3aa8fc', 'request_id': 'req-d83ff01a-c0ae-40d6-8748-6a7f88e06848'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:28.071966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e6cfe29946d49bfa9afcdb9dc3aa8fc', 'request_id': 'req-d83ff01a-c0ae-40d6-8748-6a7f88e06848'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:28.079534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e6cfe29946d49bfa9afcdb9dc3aa8fc', 'request_id': 'req-d83ff01a-c0ae-40d6-8748-6a7f88e06848'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:28.087199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:28.094776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] The token cache is being invalidate because project 8e6cfe29946d49bfa9afcdb9dc3aa8fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:15:28.095701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:28.096593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d83ff01a-c0ae-40d6-8748-6a7f88e06848 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:28.102920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1136/2268] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:15:27 2022] DELETE /identity/v3/projects/8e6cfe29946d49bfa9afcdb9dc3aa8fc => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:28.104721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3489a7-b807-4ee6-a29a-5deee8cd2cd6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.105066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3489a7-b807-4ee6-a29a-5deee8cd2cd6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.105401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3489a7-b807-4ee6-a29a-5deee8cd2cd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.127362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3d3489a7-b807-4ee6-a29a-5deee8cd2cd6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:28.128596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3d3489a7-b807-4ee6-a29a-5deee8cd2cd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:28.170901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:28.191901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1133/2269] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:28.201528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74450de4-10c8-4a4f-af22-2cbe9358dae6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.201813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74450de4-10c8-4a4f-af22-2cbe9358dae6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.202086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74450de4-10c8-4a4f-af22-2cbe9358dae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.224511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-74450de4-10c8-4a4f-af22-2cbe9358dae6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:28.225737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-74450de4-10c8-4a4f-af22-2cbe9358dae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:28.235030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1137/2270] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:28.239449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3489a7-b807-4ee6-a29a-5deee8cd2cd6 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.249910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3489a7-b807-4ee6-a29a-5deee8cd2cd6 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.255907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.256744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.257004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.257277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.258333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:15:28.258791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.261271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.277242 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1134/2271] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:15:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:15:28.282104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74450de4-10c8-4a4f-af22-2cbe9358dae6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.291794 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74450de4-10c8-4a4f-af22-2cbe9358dae6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.297830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.298427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.298707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.299014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.299685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.301601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.332554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1138/2272] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:15:28 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:28.336829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.347534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c5304-4540-4c56-aa94-03ca060a5e21 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.407668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.408290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.408560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.408831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.409498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.411463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.435438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1135/2273] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:15:28 2022] POST /identity/v3/users => generated 290 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:28.439664 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.460813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e469e3-bd8f-4eaf-96ac-5474b6f3f975 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.467320 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.468023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.468301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.468626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.469407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.471176 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.483191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1139/2274] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:28.488031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.500703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95389cd-c12a-45a0-8313-1102143d40af admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.505614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5d3222-b1a4-4cc3-b50b-fff61c5babe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.506462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5d3222-b1a4-4cc3-b50b-fff61c5babe9 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.506817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5d3222-b1a4-4cc3-b50b-fff61c5babe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.507202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5d3222-b1a4-4cc3-b50b-fff61c5babe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.507988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5d3222-b1a4-4cc3-b50b-fff61c5babe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.509695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5d3222-b1a4-4cc3-b50b-fff61c5babe9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.525668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1136/2275] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:28.526941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.537947 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792a6242-1323-4517-87fe-648c9fd1365b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.542900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.543693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.543975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.544255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] PATH_INFO: `/v3/projects/c4a4bb73716f4c7b80f33e7979f048e8/users/2f9ae32651c3479baee6663e7d0524b7/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.562425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4a4bb73716f4c7b80f33e7979f048e8, user_id=2f9ae32651c3479baee6663e7d0524b7, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.565208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.583463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1140/2276] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:28 2022] PUT /identity/v3/projects/c4a4bb73716f4c7b80f33e7979f048e8/users/2f9ae32651c3479baee6663e7d0524b7/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:28.594214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e8615-af85-4eb3-8af5-36e3dd09f538 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.594505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e8615-af85-4eb3-8af5-36e3dd09f538 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.594774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3e8615-af85-4eb3-8af5-36e3dd09f538 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.624266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0f3e8615-af85-4eb3-8af5-36e3dd09f538 None None] MFA Rules not processed for user `2f9ae32651c3479baee6663e7d0524b7`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:28.625596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0f3e8615-af85-4eb3-8af5-36e3dd09f538 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:28.691368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1137/2277] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:28 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:28.696064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.758960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1887bac3-fecc-499a-a0dc-ee36c60cc64e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.763868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.764905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.764905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.765208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.765944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.767985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.801502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1141/2278] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:15:28 2022] POST /identity/v3/projects => generated 379 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:28.803106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3e8615-af85-4eb3-8af5-36e3dd09f538 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.816213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3e8615-af85-4eb3-8af5-36e3dd09f538 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.856039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:28.886475 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.891466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.891898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.892176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.892953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.895072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.928247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1138/2279] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:15:28 2022] POST /identity/v3/users => generated 296 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:15:28.930554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.944859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6d26e6-3591-4595-a7fc-100c369f1050 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.950166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.950787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.951101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.951372 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:28.952075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:28.953684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:28.964833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1142/2280] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:28.969205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:28.980817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b56588-0842-48da-b9fa-8c38a7dbc2a9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:28.985635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:28.986312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:28.986593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:28.986872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] PATH_INFO: `/v3/projects/443b2bcdb5474c2c9baa6eec897360df/users/49cd08f7ac6e4bea8c1c4a7d2c9bfe09/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:29.002702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] RBAC: Authorizing `identity:create_grant(project_id=443b2bcdb5474c2c9baa6eec897360df, user_id=49cd08f7ac6e4bea8c1c4a7d2c9bfe09, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:29.005507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:29.020506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1139/2281] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:28 2022] PUT /identity/v3/projects/443b2bcdb5474c2c9baa6eec897360df/users/49cd08f7ac6e4bea8c1c4a7d2c9bfe09/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:15:29.024650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:29.035412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a50182a-cecb-421e-8598-058c8c9a787f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:29.094649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660b33bf-62da-4133-ab0d-738d88467599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:29.095222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b33bf-62da-4133-ab0d-738d88467599 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:29.095476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b33bf-62da-4133-ab0d-738d88467599 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:29.095770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b33bf-62da-4133-ab0d-738d88467599 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:29.096470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660b33bf-62da-4133-ab0d-738d88467599 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:29.098226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660b33bf-62da-4133-ab0d-738d88467599 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:29.108081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1143/2282] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:29 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:29.112300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:29.123596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648bd381-341a-4941-be48-aad35ee1817d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:29.128882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:29.129472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:29.129761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:29.130069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] PATH_INFO: `/v3/projects/443b2bcdb5474c2c9baa6eec897360df/users/49cd08f7ac6e4bea8c1c4a7d2c9bfe09/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:29.132167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=443b2bcdb5474c2c9baa6eec897360df, user_id=49cd08f7ac6e4bea8c1c4a7d2c9bfe09, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:29.135007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:29.147163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1140/2283] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:29 2022] PUT /identity/v3/projects/443b2bcdb5474c2c9baa6eec897360df/users/49cd08f7ac6e4bea8c1c4a7d2c9bfe09/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:29.160487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd41cb8-929c-4dea-bb42-983ddc6e71bf None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:29.160764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd41cb8-929c-4dea-bb42-983ddc6e71bf None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:29.161037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd41cb8-929c-4dea-bb42-983ddc6e71bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:29.169906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:29.191466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-2bd41cb8-929c-4dea-bb42-983ddc6e71bf None None] MFA Rules not processed for user `49cd08f7ac6e4bea8c1c4a7d2c9bfe09`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:29.193154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2bd41cb8-929c-4dea-bb42-983ddc6e71bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:29.275767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1144/2284] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:29 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:29.287765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:29.289614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:29.362102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1044d3-c6f8-4322-b0f4-697504205f6d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:29.365744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:29.366366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:29.366641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:29.366914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:29.375892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:29.377331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:29.386981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1141/2285] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:29 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:29.468673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd41cb8-929c-4dea-bb42-983ddc6e71bf None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:29.478532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd41cb8-929c-4dea-bb42-983ddc6e71bf None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:29.482219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:29.482915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:29.483283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:29.483621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:29.547165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:29.548642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:29.559284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1145/2286] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:29 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:35.119884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:35.121494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:35.161469 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:35.163101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:35.182598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ce851d-b027-467a-9146-c4f279cf074e service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:35.186301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:35.186932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:35.187253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:35.187536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:35.189333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:35.223396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8605d26b-c593-40d3-8a4e-101886c662c1 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:35.227452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795a486c-d6d1-4b36-be7a-ee2729a31228 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:35.228111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795a486c-d6d1-4b36-be7a-ee2729a31228 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:35.228394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795a486c-d6d1-4b36-be7a-ee2729a31228 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:35.228675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795a486c-d6d1-4b36-be7a-ee2729a31228 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:35.230455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-795a486c-d6d1-4b36-be7a-ee2729a31228 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:35.236270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:35.262746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:35.264237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:35.272762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:35.276571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1142/2287] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:35 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:35.303953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795a486c-d6d1-4b36-be7a-ee2729a31228 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:35.305412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795a486c-d6d1-4b36-be7a-ee2729a31228 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:35.317483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1146/2288] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:35 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:15:38.629125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:38.692392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cab8df-7800-46dc-b3c6-cf6c82e0c22d service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:38.697039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:38.697622 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:38.697902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:38.698538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] PATH_INFO: `/v3/users/2f9ae32651c3479baee6663e7d0524b7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:38.699738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f9ae32651c3479baee6663e7d0524b7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:38.701709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:38.748955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f9ae32651c3479baee6663e7d0524b7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:38.762717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f9ae32651c3479baee6663e7d0524b7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:38.770081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f9ae32651c3479baee6663e7d0524b7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:38.778925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f9ae32651c3479baee6663e7d0524b7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:38.779724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f9ae32651c3479baee6663e7d0524b7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:38.790225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1143/2289] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:15:38 2022] DELETE /identity/v3/users/2f9ae32651c3479baee6663e7d0524b7 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:38.791832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:38.792439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:38.792439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:38.812732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:38.813576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:38.883506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1147/2290] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:38.893667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:38.895163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:38.967963 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3583c2-fc55-4133-8ecb-7bc1b7f5ea11 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:38.972119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:38.973020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:38.973404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:38.973854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:38.982532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:39.004919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:39.004919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:39.005989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1144/2291] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:39.272087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:39.273593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:39.289052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e3096f-fe28-4c5d-9048-7eca217e2786 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:39.294797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:39.295823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:39.296088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:39.296359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] PATH_INFO: `/v3/projects/c4a4bb73716f4c7b80f33e7979f048e8` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:39.297615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4a4bb73716f4c7b80f33e7979f048e8)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:39.299591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:39.348264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4a4bb73716f4c7b80f33e7979f048e8', 'request_id': 'req-762e16c4-7f32-45aa-8fda-cc6caf13859e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.356565 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4a4bb73716f4c7b80f33e7979f048e8', 'request_id': 'req-762e16c4-7f32-45aa-8fda-cc6caf13859e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.363585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4a4bb73716f4c7b80f33e7979f048e8', 'request_id': 'req-762e16c4-7f32-45aa-8fda-cc6caf13859e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.380875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] The token cache is being invalidate because project c4a4bb73716f4c7b80f33e7979f048e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:15:39.381290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.382166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.383745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1148/2292] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:15:39 2022] DELETE /identity/v3/projects/c4a4bb73716f4c7b80f33e7979f048e8 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:39.388121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:39.398245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c87a5d-e32e-4d42-b35a-b8719134a53a service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:39.459944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:39.460612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:39.460896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:39.461173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] PATH_INFO: `/v3/users/49cd08f7ac6e4bea8c1c4a7d2c9bfe09` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:39.462439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=49cd08f7ac6e4bea8c1c4a7d2c9bfe09)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:39.464483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:39.516050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49cd08f7ac6e4bea8c1c4a7d2c9bfe09'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.528108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49cd08f7ac6e4bea8c1c4a7d2c9bfe09'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.534973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49cd08f7ac6e4bea8c1c4a7d2c9bfe09'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.543302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49cd08f7ac6e4bea8c1c4a7d2c9bfe09'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.544003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f2898a54-0c26-4ac8-9018-6d27fa24efb0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49cd08f7ac6e4bea8c1c4a7d2c9bfe09'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.550455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1145/2293] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:15:39 2022] DELETE /identity/v3/users/49cd08f7ac6e4bea8c1c4a7d2c9bfe09 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:39.826418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:39.829476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:39.861388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e16c4-7f32-45aa-8fda-cc6caf13859e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:39.915010 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:39.925717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:39.926666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:39.927034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:39.927592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] PATH_INFO: `/v3/projects/443b2bcdb5474c2c9baa6eec897360df` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:39.929142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=443b2bcdb5474c2c9baa6eec897360df)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:39.931160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:39.982785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '443b2bcdb5474c2c9baa6eec897360df', 'request_id': 'req-e8d9035f-5524-4141-a1ab-fb10824083e8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.991423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '443b2bcdb5474c2c9baa6eec897360df', 'request_id': 'req-e8d9035f-5524-4141-a1ab-fb10824083e8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:39.997227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '443b2bcdb5474c2c9baa6eec897360df', 'request_id': 'req-e8d9035f-5524-4141-a1ab-fb10824083e8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:40.008867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] The token cache is being invalidate because project 443b2bcdb5474c2c9baa6eec897360df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:15:40.009160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:40.009836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-e8d9035f-5524-4141-a1ab-fb10824083e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:40.011652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1149/2294] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:15:39 2022] DELETE /identity/v3/projects/443b2bcdb5474c2c9baa6eec897360df => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:40.016656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3317da8-e394-421e-b1a9-096fc7e268ff None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.017074 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3317da8-e394-421e-b1a9-096fc7e268ff None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.017213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3317da8-e394-421e-b1a9-096fc7e268ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.037084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c3317da8-e394-421e-b1a9-096fc7e268ff None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:40.038330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c3317da8-e394-421e-b1a9-096fc7e268ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:40.107100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1146/2295] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:40.111278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb68eef-61b6-4713-8837-bdc059ee7600 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.111585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb68eef-61b6-4713-8837-bdc059ee7600 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.111870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb68eef-61b6-4713-8837-bdc059ee7600 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.132456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6eb68eef-61b6-4713-8837-bdc059ee7600 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:40.133662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6eb68eef-61b6-4713-8837-bdc059ee7600 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:40.141839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1150/2296] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:40.146350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3317da8-e394-421e-b1a9-096fc7e268ff None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.156621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3317da8-e394-421e-b1a9-096fc7e268ff None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.161503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.162131 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.162415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.162724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.163780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:15:40.164216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.166708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.182988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1147/2297] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:15:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:15:40.186909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb68eef-61b6-4713-8837-bdc059ee7600 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.196920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb68eef-61b6-4713-8837-bdc059ee7600 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.201977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.202609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.202895 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.203209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.203900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.205838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.235409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1151/2298] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:15:40 2022] POST /identity/v3/projects => generated 365 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:15:40.240385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.245684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:40.252034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7149833-fca3-4ff2-99fa-62cb4d4aa446 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.308200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.308814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.309094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.309371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.310058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.312134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.335600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1148/2299] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:15:40 2022] POST /identity/v3/users => generated 283 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:15:40.340451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.355167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9125c-3777-41a0-b285-ed5c23f22e6e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.360326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.360950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.361237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.361513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.362225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.363913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.374986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1152/2300] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:40 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:15:40.379150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.389839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de871979-6db3-4966-9c63-b355893498aa admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.394832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1623ead-5a52-40f2-8c42-cb4efa488a68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.395497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1623ead-5a52-40f2-8c42-cb4efa488a68 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.395783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1623ead-5a52-40f2-8c42-cb4efa488a68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.396046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1623ead-5a52-40f2-8c42-cb4efa488a68 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.396759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1623ead-5a52-40f2-8c42-cb4efa488a68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.398401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1623ead-5a52-40f2-8c42-cb4efa488a68 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.409003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1149/2301] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:40.413332 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.423966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc696409-6c8b-4e4a-9492-ca31f005bf0a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.428823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.429483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.429764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.430040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] PATH_INFO: `/v3/projects/37df4bb037f3467aa1cccfa07d8636bd/users/0cac1174e9d64c2ba4c280cd8dd070e6/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.444727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] RBAC: Authorizing `identity:create_grant(project_id=37df4bb037f3467aa1cccfa07d8636bd, user_id=0cac1174e9d64c2ba4c280cd8dd070e6, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.447493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.462822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1153/2302] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:40 2022] PUT /identity/v3/projects/37df4bb037f3467aa1cccfa07d8636bd/users/0cac1174e9d64c2ba4c280cd8dd070e6/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:40.477920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b4a74-a541-4642-8c2f-ca4fdd7726c0 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.478207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b4a74-a541-4642-8c2f-ca4fdd7726c0 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.478480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b4a74-a541-4642-8c2f-ca4fdd7726c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.506878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4f5b4a74-a541-4642-8c2f-ca4fdd7726c0 None None] MFA Rules not processed for user `0cac1174e9d64c2ba4c280cd8dd070e6`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:40.508070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4f5b4a74-a541-4642-8c2f-ca4fdd7726c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:40.572847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1150/2303] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:40 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:40.578730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.651647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc3eb93-8391-45b8-a5db-3b152a329cce admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.660754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.661663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.662125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.662532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.663395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.665291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.696028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1154/2304] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:15:40 2022] POST /identity/v3/projects => generated 361 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:40.700879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5b4a74-a541-4642-8c2f-ca4fdd7726c0 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.712785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5b4a74-a541-4642-8c2f-ca4fdd7726c0 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.771354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.771939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.772322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.772603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.773280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.775532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.794386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:40.801228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1151/2305] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:15:40 2022] POST /identity/v3/users => generated 287 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:40.805494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.812957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:40.818754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e1ef3c-1fcd-4271-bc2e-dc3f247af0c6 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.826313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.827021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.827610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.828054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.829042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.830687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.842074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1155/2306] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:40 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:40.846486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.856936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80f875b-c81e-428e-852c-5ef0def6671b admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.862261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.862988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.863263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.863537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] PATH_INFO: `/v3/projects/2def64e66ccf45b492568facff525bac/users/5f4a2875b5eb4d37991c7ad08e6737ec/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.880325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2def64e66ccf45b492568facff525bac, user_id=5f4a2875b5eb4d37991c7ad08e6737ec, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.883071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.897697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1152/2307] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:40 2022] PUT /identity/v3/projects/2def64e66ccf45b492568facff525bac/users/5f4a2875b5eb4d37991c7ad08e6737ec/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:40.901705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:40.912344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d9fa24-4ef6-4034-8d3e-1d553a4d2193 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:40.972980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a499d8f2-3124-4edd-9f39-f6cdc498102e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:40.973602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a499d8f2-3124-4edd-9f39-f6cdc498102e admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:40.973886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a499d8f2-3124-4edd-9f39-f6cdc498102e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:40.974161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a499d8f2-3124-4edd-9f39-f6cdc498102e admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:40.974864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a499d8f2-3124-4edd-9f39-f6cdc498102e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:40.976828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a499d8f2-3124-4edd-9f39-f6cdc498102e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:40.987613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1156/2308] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:40 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:15:40.993441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:41.004186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c77aec-5cc3-4239-9e28-b1c690e01548 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:41.009808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:41.010478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:41.010751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:41.011050 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] PATH_INFO: `/v3/projects/2def64e66ccf45b492568facff525bac/users/5f4a2875b5eb4d37991c7ad08e6737ec/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:41.013113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=2def64e66ccf45b492568facff525bac, user_id=5f4a2875b5eb4d37991c7ad08e6737ec, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:41.015802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:41.027906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1153/2309] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:40 2022] PUT /identity/v3/projects/2def64e66ccf45b492568facff525bac/users/5f4a2875b5eb4d37991c7ad08e6737ec/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:41.039326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467105d9-326d-4838-bfa9-0c340f863c36 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:41.039619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467105d9-326d-4838-bfa9-0c340f863c36 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:41.039899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467105d9-326d-4838-bfa9-0c340f863c36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:41.068763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-467105d9-326d-4838-bfa9-0c340f863c36 None None] MFA Rules not processed for user `5f4a2875b5eb4d37991c7ad08e6737ec`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:41.070064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-467105d9-326d-4838-bfa9-0c340f863c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:41.150666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1157/2310] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:41 2022] POST /identity/v3/auth/tokens => generated 3295 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:15:41.162836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:41.165264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:41.227410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d30f9-84fb-4945-9ace-d48268359d2e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:41.231451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:41.232819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:41.233066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:41.233340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:41.243534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:41.244991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:41.254756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1154/2311] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:41 2022] GET /identity/v3/auth/tokens => generated 3295 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:41.335998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467105d9-326d-4838-bfa9-0c340f863c36 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:41.346385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467105d9-326d-4838-bfa9-0c340f863c36 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:41.350196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9d4ace-0158-4c08-bba3-58afdb7945f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:41.350845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9d4ace-0158-4c08-bba3-58afdb7945f8 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:41.351167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9d4ace-0158-4c08-bba3-58afdb7945f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:41.351448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9d4ace-0158-4c08-bba3-58afdb7945f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:41.413435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9d4ace-0158-4c08-bba3-58afdb7945f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:41.414919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9d4ace-0158-4c08-bba3-58afdb7945f8 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:41.424357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1158/2312] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:41 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:47.140329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:47.141999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:47.220626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31571823-40cc-4bd0-972d-9ab572f189be service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:47.227417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:47.228043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:47.228318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:47.228589 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:47.230715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:47.290684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:47.292244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:47.301632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1155/2313] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:47.567162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a898d-497f-4ad0-8963-359f47fb5a9b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:47.567679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a898d-497f-4ad0-8963-359f47fb5a9b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:47.567780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a898d-497f-4ad0-8963-359f47fb5a9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:47.587597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-470a898d-497f-4ad0-8963-359f47fb5a9b None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:47.588828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-470a898d-497f-4ad0-8963-359f47fb5a9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:47.598983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:47.649698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1159/2314] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:15:47 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:47.698576 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:47.700153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:47.705291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:47.776359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5de65d9-2578-4f4f-8657-8248f4b17685 service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:47.781461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6882c266-1ce0-47f7-bb5c-18dfe43fde91 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:47.782084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6882c266-1ce0-47f7-bb5c-18dfe43fde91 service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:47.782360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6882c266-1ce0-47f7-bb5c-18dfe43fde91 service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:47.782633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6882c266-1ce0-47f7-bb5c-18dfe43fde91 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:47.790670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6882c266-1ce0-47f7-bb5c-18dfe43fde91 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:47.792443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6882c266-1ce0-47f7-bb5c-18dfe43fde91 service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:47.800628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1156/2315] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:15:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:15:50.942262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a898d-497f-4ad0-8963-359f47fb5a9b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:51.023392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a898d-497f-4ad0-8963-359f47fb5a9b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:51.028308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:51.029166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:51.029611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:51.030047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] PATH_INFO: `/v3/users/0cac1174e9d64c2ba4c280cd8dd070e6` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:51.031551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0cac1174e9d64c2ba4c280cd8dd070e6)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:51.033677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:51.092473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0cac1174e9d64c2ba4c280cd8dd070e6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.102543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0cac1174e9d64c2ba4c280cd8dd070e6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.114077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0cac1174e9d64c2ba4c280cd8dd070e6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.148109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0cac1174e9d64c2ba4c280cd8dd070e6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.151257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0cac1174e9d64c2ba4c280cd8dd070e6'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.172239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1160/2316] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:15:50 2022] DELETE /identity/v3/users/0cac1174e9d64c2ba4c280cd8dd070e6 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:51.174063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:51.174578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:51.174683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:51.205319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:51.206569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:51.298445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1157/2317] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:51.305107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:51.307864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:51.371691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:51.396963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd026c-d1eb-42b0-8eed-fdef6e2f8f10 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:51.400738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:51.401362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:51.401636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:51.401905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:51.415172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:51.416618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:51.429554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1161/2318] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:15:51.734722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:51.736244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:51.751497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4721d7d3-7618-46e1-ac73-9c5fd7b23deb None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:51.757473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:51.758140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:51.758423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:51.758703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] PATH_INFO: `/v3/projects/37df4bb037f3467aa1cccfa07d8636bd` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:51.760487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=37df4bb037f3467aa1cccfa07d8636bd)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:51.762776 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:51.821316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37df4bb037f3467aa1cccfa07d8636bd', 'request_id': 'req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.834023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37df4bb037f3467aa1cccfa07d8636bd', 'request_id': 'req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.841721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37df4bb037f3467aa1cccfa07d8636bd', 'request_id': 'req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.853766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] The token cache is being invalidate because project 37df4bb037f3467aa1cccfa07d8636bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:15:51.854130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.855061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:51.859727 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1158/2319] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:15:51 2022] DELETE /identity/v3/projects/37df4bb037f3467aa1cccfa07d8636bd => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:51.861233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:51.871483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6001570-2406-4487-b1bd-b68301b520a8 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:51.939948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:51.940616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:51.940900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:51.941178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] PATH_INFO: `/v3/users/5f4a2875b5eb4d37991c7ad08e6737ec` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:51.942351 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f4a2875b5eb4d37991c7ad08e6737ec)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:51.944500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:52.009623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f4a2875b5eb4d37991c7ad08e6737ec'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.022241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f4a2875b5eb4d37991c7ad08e6737ec'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.036142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f4a2875b5eb4d37991c7ad08e6737ec'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.052153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f4a2875b5eb4d37991c7ad08e6737ec'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.053095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b06a5328-868a-467d-b3fd-e95e21ff173f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f4a2875b5eb4d37991c7ad08e6737ec'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.060397 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1162/2320] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:15:51 2022] DELETE /identity/v3/users/5f4a2875b5eb4d37991c7ad08e6737ec => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:15:52.426267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:52.427755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:52.449237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9f04a7-7278-4a95-9bfe-4ffb7bed6c4c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:52.476390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:52.549297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:52.550089 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:52.550396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:52.550835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] PATH_INFO: `/v3/projects/2def64e66ccf45b492568facff525bac` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:52.552108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2def64e66ccf45b492568facff525bac)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:52.554240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:52.628687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2def64e66ccf45b492568facff525bac', 'request_id': 'req-19ba50a9-d9b3-490c-8091-9843886d0df9'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.638462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2def64e66ccf45b492568facff525bac', 'request_id': 'req-19ba50a9-d9b3-490c-8091-9843886d0df9'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.646979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2def64e66ccf45b492568facff525bac', 'request_id': 'req-19ba50a9-d9b3-490c-8091-9843886d0df9'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.657995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] The token cache is being invalidate because project 2def64e66ccf45b492568facff525bac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:15:52.658619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.659498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-19ba50a9-d9b3-490c-8091-9843886d0df9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:15:52.661230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1159/2321] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:15:52 2022] DELETE /identity/v3/projects/2def64e66ccf45b492568facff525bac => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:52.667322 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d2f28-0bf8-404a-b4c8-d7e8aa7cb707 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:52.670749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d2f28-0bf8-404a-b4c8-d7e8aa7cb707 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:52.670886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d2f28-0bf8-404a-b4c8-d7e8aa7cb707 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:52.692411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ff6d2f28-0bf8-404a-b4c8-d7e8aa7cb707 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:52.693169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ff6d2f28-0bf8-404a-b4c8-d7e8aa7cb707 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:52.761393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:52.778507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1163/2322] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:52.780135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b27472-878e-4003-a69c-d6c8f401602f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:52.780509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b27472-878e-4003-a69c-d6c8f401602f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:52.780862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b27472-878e-4003-a69c-d6c8f401602f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:52.801753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-37b27472-878e-4003-a69c-d6c8f401602f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:52.803095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-37b27472-878e-4003-a69c-d6c8f401602f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:52.812171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1160/2323] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:15:52.816532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d2f28-0bf8-404a-b4c8-d7e8aa7cb707 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:52.829375 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d2f28-0bf8-404a-b4c8-d7e8aa7cb707 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:52.835544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:52.836280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:52.836629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:52.836964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:52.837765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:15:52.838126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:52.839916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:52.855303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1164/2324] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:15:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:15:52.858192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b27472-878e-4003-a69c-d6c8f401602f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:52.868609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b27472-878e-4003-a69c-d6c8f401602f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:52.873557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:52.874275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:52.874621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:52.874985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:52.875746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:52.877737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:52.914621 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1161/2325] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:15:52 2022] POST /identity/v3/projects => generated 375 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:52.918921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:52.932046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cbfeab-b219-4a46-b5fa-e0a3b7780c32 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:53.004778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:53.006133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:53.006404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:53.006686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:53.007398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:53.013485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:53.038844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1165/2326] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:15:52 2022] POST /identity/v3/users => generated 288 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:15:53.043338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:53.054783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a3b16-b80c-45a6-9671-a60a1f2b5275 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:53.059954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:53.060620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:53.060904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:53.061189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:53.061927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:53.063689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:53.076098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1162/2327] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:53.080651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:53.091932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ef3bf2-7f56-45e1-9ced-65ae2ad5f48a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:53.097939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864a182-a2cd-41c0-be81-f4a768f0ce4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:53.098599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864a182-a2cd-41c0-be81-f4a768f0ce4c admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:53.099195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864a182-a2cd-41c0-be81-f4a768f0ce4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:53.099475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864a182-a2cd-41c0-be81-f4a768f0ce4c admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:53.100216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864a182-a2cd-41c0-be81-f4a768f0ce4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:53.101941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864a182-a2cd-41c0-be81-f4a768f0ce4c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:53.117779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1166/2328] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:15:53 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:15:53.120540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:53.131916 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69ac5b-d79e-45bb-8c4e-fadbb5a8a618 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:53.136923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:53.137635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:53.137917 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:53.138381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] PATH_INFO: `/v3/projects/f2d65ff278024a2c923fba3cb9a9d1a7/users/c6dd0ef1e220423a8c4305a1c8eec32d/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:53.162552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2d65ff278024a2c923fba3cb9a9d1a7, user_id=c6dd0ef1e220423a8c4305a1c8eec32d, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:53.165423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:53.183625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1163/2329] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:15:53 2022] PUT /identity/v3/projects/f2d65ff278024a2c923fba3cb9a9d1a7/users/c6dd0ef1e220423a8c4305a1c8eec32d/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:15:53.193781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:53.194098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:53.194405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:53.259088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] MFA Rules not processed for user `c6dd0ef1e220423a8c4305a1c8eec32d`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:15:53.260684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:53.336083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1167/2330] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:15:53 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:15:53.348075 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:15:53.349927 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:15:53.403170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:15:53.419940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104eb476-4d8f-4ba5-88d4-75475af5f9f6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:15:53.423930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6606ea61-a698-4e58-ae84-b00035a7ff4b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:15:53.424563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606ea61-a698-4e58-ae84-b00035a7ff4b service glance] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:15:53.424845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606ea61-a698-4e58-ae84-b00035a7ff4b service glance] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:15:53.425120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6606ea61-a698-4e58-ae84-b00035a7ff4b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:15:53.434299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6606ea61-a698-4e58-ae84-b00035a7ff4b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:15:53.435872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6606ea61-a698-4e58-ae84-b00035a7ff4b service glance] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:15:53.445323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1164/2331] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:15:53 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:01.830996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:01.832547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:01.894764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03c9bc5-15e2-43de-ab27-df9e6fbe5b8f None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:01.899217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87354c8-7b44-4b11-9fb6-bddd4516a3d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:01.899989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87354c8-7b44-4b11-9fb6-bddd4516a3d6 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:01.900276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87354c8-7b44-4b11-9fb6-bddd4516a3d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:01.900783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87354c8-7b44-4b11-9fb6-bddd4516a3d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:01.902860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b87354c8-7b44-4b11-9fb6-bddd4516a3d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:01.913924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87354c8-7b44-4b11-9fb6-bddd4516a3d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:01.915442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87354c8-7b44-4b11-9fb6-bddd4516a3d6 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:01.919293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:01.927899 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1168/2332] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:01 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:02.042611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b708c810-ccd6-47ba-8637-7857a540974e None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.045734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b708c810-ccd6-47ba-8637-7857a540974e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.046033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b708c810-ccd6-47ba-8637-7857a540974e None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.047521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1165/2333] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:16:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:16:02.054221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d91cc-c38f-403e-bddb-24dc37fb63d0 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.054505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d91cc-c38f-403e-bddb-24dc37fb63d0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.054782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d91cc-c38f-403e-bddb-24dc37fb63d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.079478 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3a8d91cc-c38f-403e-bddb-24dc37fb63d0 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:02.080784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3a8d91cc-c38f-403e-bddb-24dc37fb63d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:02.158852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1169/2334] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:16:02 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:02.171235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b708c810-ccd6-47ba-8637-7857a540974e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:02.173094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b708c810-ccd6-47ba-8637-7857a540974e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:02.186364 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b708c810-ccd6-47ba-8637-7857a540974e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:02.189812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b9198b-7904-47fa-9398-a2c65275772c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:02.190420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9198b-7904-47fa-9398-a2c65275772c service glance] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.190701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9198b-7904-47fa-9398-a2c65275772c service glance] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.190999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9198b-7904-47fa-9398-a2c65275772c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.213539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b9198b-7904-47fa-9398-a2c65275772c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:02.215683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b9198b-7904-47fa-9398-a2c65275772c service glance] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:02.229060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1166/2335] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:02 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:16:02.310233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaab081-a3b1-4aac-b5af-326b8fe2edea None None] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.310520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaab081-a3b1-4aac-b5af-326b8fe2edea None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.311111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaab081-a3b1-4aac-b5af-326b8fe2edea None None] PATH_INFO: `/` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.312651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1170/2336] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:16:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 26 03:16:02.318396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9fbc74-9524-4763-a0d8-cac4bb67718a None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.319962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9fbc74-9524-4763-a0d8-cac4bb67718a None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.320429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9fbc74-9524-4763-a0d8-cac4bb67718a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.351497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1b9fbc74-9524-4763-a0d8-cac4bb67718a None None] MFA Rules not processed for user `29a1868eb95f4396aa3fd2da677b40f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:02.352635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1b9fbc74-9524-4763-a0d8-cac4bb67718a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:02.359473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1167/2337] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:16:02 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:02.371508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaab081-a3b1-4aac-b5af-326b8fe2edea None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:02.385149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaab081-a3b1-4aac-b5af-326b8fe2edea None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:02.388885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db46a40c-e803-4216-abc9-ddf2d6fc6b66 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:02.389516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46a40c-e803-4216-abc9-ddf2d6fc6b66 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.390182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46a40c-e803-4216-abc9-ddf2d6fc6b66 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.390182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46a40c-e803-4216-abc9-ddf2d6fc6b66 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.397872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db46a40c-e803-4216-abc9-ddf2d6fc6b66 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:02.399362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db46a40c-e803-4216-abc9-ddf2d6fc6b66 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:02.407711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1171/2338] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:02 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:02.498344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954fc01-4409-4c21-9412-a5c79c745b15 None None] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.498598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954fc01-4409-4c21-9412-a5c79c745b15 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.498936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4954fc01-4409-4c21-9412-a5c79c745b15 None None] PATH_INFO: `/` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.500347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1168/2339] 10.222.0.42 () {54 vars in 921 bytes} [Tue Jul 26 03:16:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 26 03:16:02.505828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.506113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.506394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.526464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] MFA Rules not processed for user `e2c09978fa3f44d895cbc17efdca2b32`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:02.527696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:02.537022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1172/2340] 10.222.0.42 () {58 vars in 1018 bytes} [Tue Jul 26 03:16:02 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:16:02.547749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4954fc01-4409-4c21-9412-a5c79c745b15 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:02.559412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4954fc01-4409-4c21-9412-a5c79c745b15 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:02.572157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:02.573217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.573553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.573902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.585104 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:02.586538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:02.597038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1169/2341] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:02 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:02.666098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:02.668652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:02.750566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22146b2c-e7d5-4089-880f-613666ea48b3 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:02.756274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ab4dfb-09ff-4923-9362-57d741ca806c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:02.757012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ab4dfb-09ff-4923-9362-57d741ca806c service placement] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:02.757327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ab4dfb-09ff-4923-9362-57d741ca806c service placement] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:02.757627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ab4dfb-09ff-4923-9362-57d741ca806c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:02.760110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-14ab4dfb-09ff-4923-9362-57d741ca806c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:02.772306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ab4dfb-09ff-4923-9362-57d741ca806c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:02.773903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ab4dfb-09ff-4923-9362-57d741ca806c service placement] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:02.784203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1173/2342] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:02 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:03.611552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:03.640983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7162cf-8091-44fa-8070-d916f102bb8a service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:03.646450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:03.646979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:03.647283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:03.647563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:03.649892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:03.660780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:03.662857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:03.674878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1170/2343] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:03 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:04.584873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca90b02-3bba-4fbe-baef-8df23c133165 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:04.585422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca90b02-3bba-4fbe-baef-8df23c133165 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:04.585422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca90b02-3bba-4fbe-baef-8df23c133165 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:04.608126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6ca90b02-3bba-4fbe-baef-8df23c133165 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:04.610096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6ca90b02-3bba-4fbe-baef-8df23c133165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:04.684949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1174/2344] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:16:04 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:04.697126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:04.698918 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:04.769170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8ec32a-1245-46d5-bbda-692ea6f48585 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:04.774136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d5255-7316-4c34-95c0-a7736a807515 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:04.774757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d5255-7316-4c34-95c0-a7736a807515 service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:04.775070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d5255-7316-4c34-95c0-a7736a807515 service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:04.775343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d5255-7316-4c34-95c0-a7736a807515 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:04.783378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d5255-7316-4c34-95c0-a7736a807515 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:04.784959 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d5255-7316-4c34-95c0-a7736a807515 service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:04.792833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1171/2345] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:16:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 26 03:16:11.403104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca90b02-3bba-4fbe-baef-8df23c133165 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:11.409225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:11.472412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca90b02-3bba-4fbe-baef-8df23c133165 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:11.482580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:11.483599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:11.483872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:11.484145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] PATH_INFO: `/v3/users/c6dd0ef1e220423a8c4305a1c8eec32d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:11.485334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6dd0ef1e220423a8c4305a1c8eec32d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:11.487401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:11.543762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6dd0ef1e220423a8c4305a1c8eec32d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:11.554598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6dd0ef1e220423a8c4305a1c8eec32d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:11.562903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6dd0ef1e220423a8c4305a1c8eec32d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:11.575365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6dd0ef1e220423a8c4305a1c8eec32d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:11.576031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6dd0ef1e220423a8c4305a1c8eec32d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:11.582544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1175/2346] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:16:11 2022] DELETE /identity/v3/users/c6dd0ef1e220423a8c4305a1c8eec32d => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:11.587616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:11.588178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:11.588178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:11.610699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:11.611987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:11.680197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1172/2347] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:16:11.690507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:11.691982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:11.761023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1614af35-01af-4db7-a277-359d5cf7eb17 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:11.766223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f6078-ee4b-41b0-9a04-95232f309708 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:11.767197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f6078-ee4b-41b0-9a04-95232f309708 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:11.767841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f6078-ee4b-41b0-9a04-95232f309708 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:11.768305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f6078-ee4b-41b0-9a04-95232f309708 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:11.780039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3f6078-ee4b-41b0-9a04-95232f309708 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:11.781778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3f6078-ee4b-41b0-9a04-95232f309708 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:11.811567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1176/2348] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:12.121929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.123633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:12.136604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab0ba5f-0ae2-4682-ab9c-4128c247e554 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.142436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.143143 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.143425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.143700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] PATH_INFO: `/v3/projects/f2d65ff278024a2c923fba3cb9a9d1a7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.145071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2d65ff278024a2c923fba3cb9a9d1a7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.147055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.179051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:12.214106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2d65ff278024a2c923fba3cb9a9d1a7', 'request_id': 'req-b18b9972-8350-424a-88a7-be1e9c755aac'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:12.225817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2d65ff278024a2c923fba3cb9a9d1a7', 'request_id': 'req-b18b9972-8350-424a-88a7-be1e9c755aac'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:12.239894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2d65ff278024a2c923fba3cb9a9d1a7', 'request_id': 'req-b18b9972-8350-424a-88a7-be1e9c755aac'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:12.256860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] The token cache is being invalidate because project f2d65ff278024a2c923fba3cb9a9d1a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:16:12.257149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:12.257938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b18b9972-8350-424a-88a7-be1e9c755aac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:12.263004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1173/2349] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:16:12 2022] DELETE /identity/v3/projects/f2d65ff278024a2c923fba3cb9a9d1a7 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:12.265294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c16fa09-e53d-4b44-9d97-e81485e68d57 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.265633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c16fa09-e53d-4b44-9d97-e81485e68d57 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.266007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c16fa09-e53d-4b44-9d97-e81485e68d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.303356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-9c16fa09-e53d-4b44-9d97-e81485e68d57 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:12.304771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9c16fa09-e53d-4b44-9d97-e81485e68d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:12.332609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:12.378093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1177/2350] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:12.386156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3450fe-8e88-4eb0-aea8-3a7f1408aa4e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.386574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3450fe-8e88-4eb0-aea8-3a7f1408aa4e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.386998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3450fe-8e88-4eb0-aea8-3a7f1408aa4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.408243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ff3450fe-8e88-4eb0-aea8-3a7f1408aa4e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:12.409581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ff3450fe-8e88-4eb0-aea8-3a7f1408aa4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:12.419167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1174/2351] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:12.422413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c16fa09-e53d-4b44-9d97-e81485e68d57 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.432531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c16fa09-e53d-4b44-9d97-e81485e68d57 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.437607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.438256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.438651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.438925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.439723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:16:12.440215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.442000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.456212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1178/2352] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:16:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:16:12.460670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3450fe-8e88-4eb0-aea8-3a7f1408aa4e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.471066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3450fe-8e88-4eb0-aea8-3a7f1408aa4e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.476574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.477286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.477651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.478274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.479117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.481357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.512940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1175/2353] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:16:12 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:16:12.517265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.528268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69838178-d669-4505-95de-6c158ee06edb admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.588838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.589570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.589916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.590253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.591169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.593263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.622308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1179/2354] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:16:12 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:16:12.624021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.636885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e0231c-e4eb-4bb4-9d07-bfea905f5795 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.642513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.643158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.643441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.643713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.644430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.646126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.657039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1176/2355] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:16:12.661174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.672236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7894e621-7898-409e-9679-c4975f6335c0 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.684995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee75f04f-f648-43aa-abdf-1851cd0ab474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.685887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee75f04f-f648-43aa-abdf-1851cd0ab474 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.686342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee75f04f-f648-43aa-abdf-1851cd0ab474 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.686636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee75f04f-f648-43aa-abdf-1851cd0ab474 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.687545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee75f04f-f648-43aa-abdf-1851cd0ab474 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.689282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee75f04f-f648-43aa-abdf-1851cd0ab474 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.702763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1180/2356] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:12 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:12.707220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.718103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3353579-d2fa-44b5-9445-212599444e42 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.722757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.723458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.723739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.724014 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] PATH_INFO: `/v3/projects/8bd77269b8504984b3e086be66ac9713/users/c70c080fc8354ccf90391984baf6052f/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.739230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bd77269b8504984b3e086be66ac9713, user_id=c70c080fc8354ccf90391984baf6052f, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.742419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.758485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1177/2357] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:16:12 2022] PUT /identity/v3/projects/8bd77269b8504984b3e086be66ac9713/users/c70c080fc8354ccf90391984baf6052f/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:12.763721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9b41b6-9025-4cef-a669-f7c33de90e9f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.764045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9b41b6-9025-4cef-a669-f7c33de90e9f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.764323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9b41b6-9025-4cef-a669-f7c33de90e9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.783618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ee9b41b6-9025-4cef-a669-f7c33de90e9f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:12.784758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ee9b41b6-9025-4cef-a669-f7c33de90e9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:12.847253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1181/2358] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:12.857435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:12.861083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:12.963987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9299f5-ece2-46d7-abe3-c90dceb75107 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:12.967908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:12.968556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:12.968896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:12.969185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:12.979398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:12.980856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:12.994938 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1178/2359] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:13.755360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45791b77-d986-4288-be24-a9b8b94a3502 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:13.755962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45791b77-d986-4288-be24-a9b8b94a3502 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:13.755962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45791b77-d986-4288-be24-a9b8b94a3502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:13.786395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-45791b77-d986-4288-be24-a9b8b94a3502 None None] MFA Rules not processed for user `c70c080fc8354ccf90391984baf6052f`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:13.787629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-45791b77-d986-4288-be24-a9b8b94a3502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:13.809602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:13.861325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1182/2360] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:13.872485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:13.874020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:13.936707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56bd51df-88db-4fcd-a9a1-485724f7d960 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:13.940424 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:13.941034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:13.941310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:13.941580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:13.946774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:13.951896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:13.953615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:13.964273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1179/2361] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:13 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:15.419225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:15.419841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:15.419841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:15.439604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] MFA Rules not processed for user `c70c080fc8354ccf90391984baf6052f`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:15.440996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:15.451153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1183/2362] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:15.682583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:15.684192 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:15.696265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8f9867-f3e4-49fd-b85e-5f7748433122 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:15.700501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:15.701110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:15.701366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:15.701659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:15.703614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:15.713405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:15.715410 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:15.725839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1180/2363] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:15 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:17.281501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:17.282896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:17.382034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56fcc2a-823b-4cf9-ba03-f2132ad2338d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:17.387816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:17.388446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:17.388725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:17.388995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:17.390798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:17.458669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:17.460374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:17.470021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1184/2364] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:17 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:18.642425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:18.658350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8410fc-55b7-4b76-aa56-395a0c039a2c service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:18.664215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59769bc-3296-43f1-a996-2fb2153d317f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:18.665028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59769bc-3296-43f1-a996-2fb2153d317f service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:18.665312 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59769bc-3296-43f1-a996-2fb2153d317f service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:18.665591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59769bc-3296-43f1-a996-2fb2153d317f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:18.667480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e59769bc-3296-43f1-a996-2fb2153d317f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:18.680194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59769bc-3296-43f1-a996-2fb2153d317f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:18.681797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59769bc-3296-43f1-a996-2fb2153d317f service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:18.694169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1181/2365] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:18 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:21.577333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:21.588705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2749c4e-7b1f-4500-9d6c-6b85561186da service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:21.594778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:21.595432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:21.595713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:21.595992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:21.597961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:21.647645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:21.659055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:21.660753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:21.670673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1185/2366] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:22.016962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:22.017405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:22.017723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:22.049669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:22.051679 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:22.111638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1182/2367] 10.222.0.42 () {58 vars in 967 bytes} [Tue Jul 26 03:16:22 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:22.127031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:22.129004 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:22.237425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f89d700-6bd2-4ce6-8e90-07509de42d2e service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:22.244282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:22.256293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:22.256293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:22.256293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:22.258169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:22.260045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:22.269780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1186/2368] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:16:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:16:23.848138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:23.850400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:23.868343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5050415f-3f5f-4f52-941d-9fa0e6558293 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:23.886395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:23.887357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:23.887623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:23.887875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:23.890281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:23.904210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:23.905871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:23.918023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1183/2369] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:23 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:32.265951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=36 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:33.913865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=46 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:50.532894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:50.546506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90790430-a506-4c72-a258-771a450a3d9b service swift] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:50.551422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:50.552107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:50.552387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:50.552657 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:50.554802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:50.568315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:50.569762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:50.580550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1187/2370] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:53.691830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:53.693516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:53.759215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:53.760800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:53.777057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e3907d-2d9f-4b54-b619-7f09993a85e6 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:53.781748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:53.782494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:53.782835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:53.783201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] PATH_INFO: `/v3/users/9d3fd2b3c6cd47019e15fbf959d47381` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:53.783561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36c768a-c61d-419f-bf2f-f5a245ff4d2c service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:53.784499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d3fd2b3c6cd47019e15fbf959d47381)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:53.786615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:53.789265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:53.790353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:53.790637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:53.791157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:53.793583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:53.813272 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:53.814931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:53.831056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1184/2371] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:53 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:53.846732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d3fd2b3c6cd47019e15fbf959d47381'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:53.854110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d3fd2b3c6cd47019e15fbf959d47381'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:53.863163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d3fd2b3c6cd47019e15fbf959d47381'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:53.876270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d3fd2b3c6cd47019e15fbf959d47381'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:53.876913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c7b5b325-5a4d-43aa-8ed8-6818efcbc296 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d3fd2b3c6cd47019e15fbf959d47381'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:53.882896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1188/2372] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:16:53 2022] DELETE /identity/v3/users/9d3fd2b3c6cd47019e15fbf959d47381 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:16:54.063509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.074848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8a422f-3e80-4042-a891-d4acc98806ab service placement] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.134419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.135105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.135388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.135664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] PATH_INFO: `/v3/projects/9d8d482c37d849cf8bdf035e110e0f34` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.136852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d8d482c37d849cf8bdf035e110e0f34)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.138774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.185108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d8d482c37d849cf8bdf035e110e0f34', 'request_id': 'req-6383117b-1645-4f50-b614-f370bbca864a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:54.193511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d8d482c37d849cf8bdf035e110e0f34', 'request_id': 'req-6383117b-1645-4f50-b614-f370bbca864a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:54.199867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d8d482c37d849cf8bdf035e110e0f34', 'request_id': 'req-6383117b-1645-4f50-b614-f370bbca864a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:54.211037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] The token cache is being invalidate because project 9d8d482c37d849cf8bdf035e110e0f34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:16:54.211338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:54.212084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6383117b-1645-4f50-b614-f370bbca864a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:54.213794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1185/2373] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:16:54 2022] DELETE /identity/v3/projects/9d8d482c37d849cf8bdf035e110e0f34 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:54.218888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c31db2c-df62-4b05-898c-862803df4b01 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.219208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c31db2c-df62-4b05-898c-862803df4b01 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.219485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c31db2c-df62-4b05-898c-862803df4b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.238795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-4c31db2c-df62-4b05-898c-862803df4b01 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:54.240484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4c31db2c-df62-4b05-898c-862803df4b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:54.306929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1189/2374] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:16:54.316190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d81eac0-d174-4fe9-81f5-2116386ed766 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.316618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d81eac0-d174-4fe9-81f5-2116386ed766 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.316929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d81eac0-d174-4fe9-81f5-2116386ed766 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.338120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1d81eac0-d174-4fe9-81f5-2116386ed766 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:54.339390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1d81eac0-d174-4fe9-81f5-2116386ed766 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:54.348602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1186/2375] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:54.352265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c31db2c-df62-4b05-898c-862803df4b01 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.362274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c31db2c-df62-4b05-898c-862803df4b01 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.366912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.367632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.367817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.368127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.368826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:16:54.369223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.370858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.383115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1190/2376] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:16:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:16:54.387714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d81eac0-d174-4fe9-81f5-2116386ed766 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.397587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d81eac0-d174-4fe9-81f5-2116386ed766 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.402706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.403497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.403765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.404061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.404751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.406655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.418713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:54.445928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1187/2377] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:16:54 2022] POST /identity/v3/projects => generated 405 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:16:54.450675 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.461649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599c74e1-ca42-4753-b05f-337a2112b257 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.506817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:54.521935 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.522554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.522829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.523131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.523792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.525717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.547320 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1191/2378] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:16:54 2022] POST /identity/v3/users => generated 303 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:16:54.551549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.561826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd22d3-c392-4293-af53-1ae8b0d49810 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.566719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.567433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.567719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.567988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.568691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.570306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.581156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1188/2379] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:54.585287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.596819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f773d-dddc-4001-8a17-beb306906675 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.602151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d90e82-f1c3-4439-9894-bb2cdeeba927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.602812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d90e82-f1c3-4439-9894-bb2cdeeba927 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.603088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d90e82-f1c3-4439-9894-bb2cdeeba927 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.603723 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d90e82-f1c3-4439-9894-bb2cdeeba927 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.604593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d90e82-f1c3-4439-9894-bb2cdeeba927 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.605852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d90e82-f1c3-4439-9894-bb2cdeeba927 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.618105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1192/2380] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:54 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:54.622381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.633027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62cd79c7-c728-419a-914c-31252cc08078 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.638534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.639254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.639538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.639815 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] PATH_INFO: `/v3/projects/6b1c93ffc9704f20a8b3ae3418268004/users/1322dda13e1b49b28ddcbcdb000fa479/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.654851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b1c93ffc9704f20a8b3ae3418268004, user_id=1322dda13e1b49b28ddcbcdb000fa479, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.657568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.672165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1189/2381] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:16:54 2022] PUT /identity/v3/projects/6b1c93ffc9704f20a8b3ae3418268004/users/1322dda13e1b49b28ddcbcdb000fa479/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:16:54.683486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38542607-ab1b-426c-8786-4a419fa733aa None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.683827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38542607-ab1b-426c-8786-4a419fa733aa None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.684222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38542607-ab1b-426c-8786-4a419fa733aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.716532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-38542607-ab1b-426c-8786-4a419fa733aa None None] MFA Rules not processed for user `1322dda13e1b49b28ddcbcdb000fa479`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:54.717758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-38542607-ab1b-426c-8786-4a419fa733aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:54.782089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1193/2382] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:54 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:16:54.786701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.853205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e459ed7-40db-413b-b2a5-149ac4999c0d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.858845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.859498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.859778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.860049 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.860718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.862618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:54.897720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1190/2383] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:16:54 2022] POST /identity/v3/projects => generated 403 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:16:54.902142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38542607-ab1b-426c-8786-4a419fa733aa None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:54.915385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38542607-ab1b-426c-8786-4a419fa733aa None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:54.983468 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:54.984271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:54.984638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:54.985021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:54.985955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:54.988845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.014557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1194/2384] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:16:54 2022] POST /identity/v3/users => generated 308 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:16:55.018689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:55.029640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50c3154-5240-4bd1-a0c9-0b25685adcf6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:55.035203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:55.035844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.036128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.036404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.037116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:55.038754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.049813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1191/2385] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:55 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:55.054142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:55.068995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45eb4d-143c-4085-be2b-9a2d572ac913 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:55.078307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:55.079399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.079795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.080202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] PATH_INFO: `/v3/projects/e84f752f51a34ffeaa4d3699823bdd0e/users/175d17d447c7434b8438d9992f88fa8b/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.098582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e84f752f51a34ffeaa4d3699823bdd0e, user_id=175d17d447c7434b8438d9992f88fa8b, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:55.101280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.116630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1195/2386] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:16:55 2022] PUT /identity/v3/projects/e84f752f51a34ffeaa4d3699823bdd0e/users/175d17d447c7434b8438d9992f88fa8b/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:55.120722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:55.130961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c664327f-79d2-41ca-b5e1-536bc63b7df9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:55.188304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93d7a95-7a9c-47e5-b61e-bf8b33fdb034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:55.188925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93d7a95-7a9c-47e5-b61e-bf8b33fdb034 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.189204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93d7a95-7a9c-47e5-b61e-bf8b33fdb034 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.189480 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93d7a95-7a9c-47e5-b61e-bf8b33fdb034 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.190178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93d7a95-7a9c-47e5-b61e-bf8b33fdb034 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:55.191910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93d7a95-7a9c-47e5-b61e-bf8b33fdb034 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.202868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1192/2387] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:55 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:55.207258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:55.217625 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e530a9-5ce3-481f-bfa2-c19e1d829f96 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:55.225135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:55.226217 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.226668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.227075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] PATH_INFO: `/v3/projects/e84f752f51a34ffeaa4d3699823bdd0e/users/175d17d447c7434b8438d9992f88fa8b/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.234680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e84f752f51a34ffeaa4d3699823bdd0e, user_id=175d17d447c7434b8438d9992f88fa8b, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:55.237473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.250285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1196/2388] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:16:55 2022] PUT /identity/v3/projects/e84f752f51a34ffeaa4d3699823bdd0e/users/175d17d447c7434b8438d9992f88fa8b/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:55.262329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcab836-e6ea-4aef-b156-86e9ef21d99b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.262617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcab836-e6ea-4aef-b156-86e9ef21d99b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.262899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcab836-e6ea-4aef-b156-86e9ef21d99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.296280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5bcab836-e6ea-4aef-b156-86e9ef21d99b None None] MFA Rules not processed for user `175d17d447c7434b8438d9992f88fa8b`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:55.297617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5bcab836-e6ea-4aef-b156-86e9ef21d99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:55.316591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:55.376645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1193/2389] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:55 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:55.387877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:55.389941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:55.400806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:55.451424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d80820-c425-46b1-b12e-0c25a9870c15 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:55.455477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f88f5dc-8c9c-4ab6-b8da-5d78936b1cf9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:55.456111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f88f5dc-8c9c-4ab6-b8da-5d78936b1cf9 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.456413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f88f5dc-8c9c-4ab6-b8da-5d78936b1cf9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.456662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f88f5dc-8c9c-4ab6-b8da-5d78936b1cf9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.466473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f88f5dc-8c9c-4ab6-b8da-5d78936b1cf9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:55.467964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f88f5dc-8c9c-4ab6-b8da-5d78936b1cf9 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.477702 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1197/2390] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:55 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:55.638301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcab836-e6ea-4aef-b156-86e9ef21d99b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:55.707223 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcab836-e6ea-4aef-b156-86e9ef21d99b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:55.712165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:55.712983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.713259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.713654 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] PATH_INFO: `/v3/users/1322dda13e1b49b28ddcbcdb000fa479` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.714826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1322dda13e1b49b28ddcbcdb000fa479)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:55.716744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.769585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1322dda13e1b49b28ddcbcdb000fa479'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:55.782339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1322dda13e1b49b28ddcbcdb000fa479'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:55.790830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1322dda13e1b49b28ddcbcdb000fa479'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:55.800072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1322dda13e1b49b28ddcbcdb000fa479'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:55.800870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1322dda13e1b49b28ddcbcdb000fa479'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:55.807865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1194/2391] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:16:55 2022] DELETE /identity/v3/users/1322dda13e1b49b28ddcbcdb000fa479 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:55.814403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.814403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.814403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.834976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:55.836209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:55.896406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1198/2392] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:55.906340 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:55.907842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:55.942814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:55.968826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9450af2-a75e-4238-ade0-478bd49e7548 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:55.972901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:55.973534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:55.973813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:55.974090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:55.983782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:55.985226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:55.994978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1195/2393] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:56.275057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:56.276602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:56.289342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badb18ab-f343-4e35-9598-d20a07b2b8d2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:56.295370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:56.296070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:56.296374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:56.296656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] PATH_INFO: `/v3/projects/6b1c93ffc9704f20a8b3ae3418268004` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:56.298394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b1c93ffc9704f20a8b3ae3418268004)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:56.300357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:56.349611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b1c93ffc9704f20a8b3ae3418268004', 'request_id': 'req-82e17e03-418f-4fbc-8cd0-db438d6de2ac'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.357038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b1c93ffc9704f20a8b3ae3418268004', 'request_id': 'req-82e17e03-418f-4fbc-8cd0-db438d6de2ac'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.362652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b1c93ffc9704f20a8b3ae3418268004', 'request_id': 'req-82e17e03-418f-4fbc-8cd0-db438d6de2ac'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.374097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] The token cache is being invalidate because project 6b1c93ffc9704f20a8b3ae3418268004 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:16:56.374381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.375120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.376733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1199/2394] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:16:56 2022] DELETE /identity/v3/projects/6b1c93ffc9704f20a8b3ae3418268004 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:56.380836 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:56.390833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39211c8-8c5a-427e-8987-2a83368bff01 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:56.449727 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:56.450419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:56.450789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:56.451184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] PATH_INFO: `/v3/users/175d17d447c7434b8438d9992f88fa8b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:56.452652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] RBAC: Authorizing `identity:delete_user(user_id=175d17d447c7434b8438d9992f88fa8b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:56.454971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:56.501885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '175d17d447c7434b8438d9992f88fa8b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.514105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '175d17d447c7434b8438d9992f88fa8b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.521079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '175d17d447c7434b8438d9992f88fa8b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.528588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '175d17d447c7434b8438d9992f88fa8b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.529372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8170bc43-9775-47c6-84d1-6f700fdcf201 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '175d17d447c7434b8438d9992f88fa8b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.538670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1196/2395] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:16:56 2022] DELETE /identity/v3/users/175d17d447c7434b8438d9992f88fa8b => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:16:56.815082 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:56.816649 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:56.832920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e17e03-418f-4fbc-8cd0-db438d6de2ac admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:56.865602 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:56.903283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:56.904264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:56.904754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:56.905043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] PATH_INFO: `/v3/projects/e84f752f51a34ffeaa4d3699823bdd0e` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:56.907115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e84f752f51a34ffeaa4d3699823bdd0e)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:56.909792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:56.961633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e84f752f51a34ffeaa4d3699823bdd0e', 'request_id': 'req-df201d57-1f59-4c52-b605-c0e1b8b217f3'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.970840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e84f752f51a34ffeaa4d3699823bdd0e', 'request_id': 'req-df201d57-1f59-4c52-b605-c0e1b8b217f3'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.976620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e84f752f51a34ffeaa4d3699823bdd0e', 'request_id': 'req-df201d57-1f59-4c52-b605-c0e1b8b217f3'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.987726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] The token cache is being invalidate because project e84f752f51a34ffeaa4d3699823bdd0e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:16:56.988013 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.988658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-df201d57-1f59-4c52-b605-c0e1b8b217f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:16:56.990190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1200/2396] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:16:56 2022] DELETE /identity/v3/projects/e84f752f51a34ffeaa4d3699823bdd0e => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:56.995321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b03547-0728-4a6b-818c-5a5a36a7ba9e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:56.995830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b03547-0728-4a6b-818c-5a5a36a7ba9e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:56.995830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b03547-0728-4a6b-818c-5a5a36a7ba9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.016724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a7b03547-0728-4a6b-818c-5a5a36a7ba9e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:57.017943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a7b03547-0728-4a6b-818c-5a5a36a7ba9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:57.065539 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:57.079775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1197/2397] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:16:57.088878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46951e71-4bda-4177-a358-14f69f750d58 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.089156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46951e71-4bda-4177-a358-14f69f750d58 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.089429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46951e71-4bda-4177-a358-14f69f750d58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.109616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-46951e71-4bda-4177-a358-14f69f750d58 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:57.110817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-46951e71-4bda-4177-a358-14f69f750d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:57.120022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1201/2398] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:57.126173 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b03547-0728-4a6b-818c-5a5a36a7ba9e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.136134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b03547-0728-4a6b-818c-5a5a36a7ba9e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.141739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.142352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.142629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.143288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.144016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:16:57.144324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.145934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.157923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1198/2399] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:16:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:16:57.162552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46951e71-4bda-4177-a358-14f69f750d58 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.176918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46951e71-4bda-4177-a358-14f69f750d58 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.183441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.184047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.184328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.184600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.185277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.187219 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.217146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1202/2400] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:16:57 2022] POST /identity/v3/projects => generated 369 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:16:57.221337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.231317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63354186-09f1-4acd-acf3-f9b82c100965 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.287520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.288127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.288403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.288674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.289350 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.291323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.316041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1199/2401] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:16:57 2022] POST /identity/v3/users => generated 285 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:16:57.320226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.331099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1674faae-cd29-429c-966f-e1c7086ddd8d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.336551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.337168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.337445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.337712 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.338497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.340206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.350454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1203/2402] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:57.354667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.364946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401223b5-3e98-40a5-90f0-e384d84bb5b4 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.370203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f761a64-c506-4bac-b023-e075f232ae58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.370808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f761a64-c506-4bac-b023-e075f232ae58 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.371130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f761a64-c506-4bac-b023-e075f232ae58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.371396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f761a64-c506-4bac-b023-e075f232ae58 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.372093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f761a64-c506-4bac-b023-e075f232ae58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.373684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f761a64-c506-4bac-b023-e075f232ae58 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.384434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1200/2403] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:57.388626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.399105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f23d2e4-8c7b-486a-bbac-be293e656bfe admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.405029 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.405692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.405969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.406238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] PATH_INFO: `/v3/projects/0b2264593b734d79949d337a7ea254bb/users/afedf95a459a4958af4feae1e52ec2a0/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.421499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b2264593b734d79949d337a7ea254bb, user_id=afedf95a459a4958af4feae1e52ec2a0, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.424197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.438819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1204/2404] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:16:57 2022] PUT /identity/v3/projects/0b2264593b734d79949d337a7ea254bb/users/afedf95a459a4958af4feae1e52ec2a0/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:57.449568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ffaf0-3cad-4d28-9c23-300def9a5057 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.449850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ffaf0-3cad-4d28-9c23-300def9a5057 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.450121 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ffaf0-3cad-4d28-9c23-300def9a5057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.477988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-638ffaf0-3cad-4d28-9c23-300def9a5057 None None] MFA Rules not processed for user `afedf95a459a4958af4feae1e52ec2a0`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:57.479231 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-638ffaf0-3cad-4d28-9c23-300def9a5057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:57.544661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1201/2405] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:57 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:57.550893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.615128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866bf134-5708-408f-82df-b7e4c41124d5 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.620213 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.620826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.621109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.621379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.622052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.624015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.628728 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:57.655237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1205/2406] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:16:57 2022] POST /identity/v3/projects => generated 367 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:16:57.660641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638ffaf0-3cad-4d28-9c23-300def9a5057 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.671303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638ffaf0-3cad-4d28-9c23-300def9a5057 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.716183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:57.728964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.729521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.729816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.730078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.730778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.732759 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.755330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1202/2407] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:16:57 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:16:57.759464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.769865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef83048e-b2ce-4237-9bad-057313c8cdab admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.775133 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.775858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.776138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.776411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.777096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.778744 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.788662 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1206/2408] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:57.792981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.803758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c4a1af-c617-42de-b9d3-78ce60a722c1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.808532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.809202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.809494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.809758 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] PATH_INFO: `/v3/projects/af9b7e8ea24e469d95ebc5beb801107d/users/d4f3d98eef94428f9d8179d9f2c47424/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.826699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af9b7e8ea24e469d95ebc5beb801107d, user_id=d4f3d98eef94428f9d8179d9f2c47424, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.829476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.844792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1203/2409] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:16:57 2022] PUT /identity/v3/projects/af9b7e8ea24e469d95ebc5beb801107d/users/d4f3d98eef94428f9d8179d9f2c47424/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:16:57.848822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.859126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec9fbb14-efda-481f-b873-1b8f668737fc admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.927067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029a4a19-830a-414a-a792-b8974ed96d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.927697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a4a19-830a-414a-a792-b8974ed96d5c admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.927977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a4a19-830a-414a-a792-b8974ed96d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.928256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a4a19-830a-414a-a792-b8974ed96d5c admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.929093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029a4a19-830a-414a-a792-b8974ed96d5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.930731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029a4a19-830a-414a-a792-b8974ed96d5c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.940839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1207/2410] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:16:57 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:16:57.945344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:57.956534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc50e02f-221a-41bd-ba87-596861ae56b3 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:57.961660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:57.962342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.962616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.963027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] PATH_INFO: `/v3/projects/af9b7e8ea24e469d95ebc5beb801107d/users/d4f3d98eef94428f9d8179d9f2c47424/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:57.965052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af9b7e8ea24e469d95ebc5beb801107d, user_id=d4f3d98eef94428f9d8179d9f2c47424, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:57.968097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:57.978376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1204/2411] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:16:57 2022] PUT /identity/v3/projects/af9b7e8ea24e469d95ebc5beb801107d/users/d4f3d98eef94428f9d8179d9f2c47424/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:16:57.995337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:57.995529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:57.995791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:58.026308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] MFA Rules not processed for user `d4f3d98eef94428f9d8179d9f2c47424`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:16:58.027626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:58.102295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1208/2412] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:16:57 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:16:58.113889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:58.115872 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:58.178314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6579b638-d10d-42fb-88a5-ff6e44ddf669 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:58.182099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:58.182715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:58.182984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:58.183295 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:58.192366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:58.193798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:58.203671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1205/2413] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:58 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:16:58.259294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:16:58.260894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:16:58.273589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133e2934-e6f6-4596-89c2-92d8a88098db None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:16:58.277609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09094063-c4d0-45f1-8f33-55150f497094 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:16:58.278225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09094063-c4d0-45f1-8f33-55150f497094 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:16:58.278505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09094063-c4d0-45f1-8f33-55150f497094 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:16:58.278775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09094063-c4d0-45f1-8f33-55150f497094 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:16:58.333149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:16:58.337697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09094063-c4d0-45f1-8f33-55150f497094 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:16:58.339310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09094063-c4d0-45f1-8f33-55150f497094 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:16:58.349391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1209/2414] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:16:58 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:01.869128 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:01.933555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d34314-8b8e-47f9-9256-fd9719b2c978 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:01.941118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:01.941789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:01.942070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:01.942345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] PATH_INFO: `/v3/users/afedf95a459a4958af4feae1e52ec2a0` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:01.943577 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afedf95a459a4958af4feae1e52ec2a0)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:01.945516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:01.998285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:02.001150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afedf95a459a4958af4feae1e52ec2a0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.022674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afedf95a459a4958af4feae1e52ec2a0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.030798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afedf95a459a4958af4feae1e52ec2a0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.038763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afedf95a459a4958af4feae1e52ec2a0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.039401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afedf95a459a4958af4feae1e52ec2a0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.045406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1206/2415] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:01 2022] DELETE /identity/v3/users/afedf95a459a4958af4feae1e52ec2a0 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:02.051786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:02.052273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:02.052370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:02.080809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:02.082022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:02.145869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1210/2416] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:02.155780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:02.157201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:02.217657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59e8fb8-a5fe-4f33-ac4c-67f8264c02b5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:02.221920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:02.222544 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:02.222825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:02.223137 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:02.233101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:02.234523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:02.244219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1207/2417] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:02.525439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:02.527108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:02.539362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a0809e-63e1-4b69-9e8b-24f34ec4c013 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:02.544195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:02.544782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:02.545056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:02.545334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] PATH_INFO: `/v3/projects/0b2264593b734d79949d337a7ea254bb` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:02.546479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b2264593b734d79949d337a7ea254bb)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:02.548427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:02.598829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b2264593b734d79949d337a7ea254bb', 'request_id': 'req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.606960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b2264593b734d79949d337a7ea254bb', 'request_id': 'req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.612802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b2264593b734d79949d337a7ea254bb', 'request_id': 'req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.624118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] The token cache is being invalidate because project 0b2264593b734d79949d337a7ea254bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:02.624408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.625049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.626598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1211/2418] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:02 2022] DELETE /identity/v3/projects/0b2264593b734d79949d337a7ea254bb => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:02.630724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:02.640434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363a242-eb22-4b48-8099-b5554f4108c7 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:02.705913 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:02.706547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:02.706828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:02.707140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] PATH_INFO: `/v3/users/d4f3d98eef94428f9d8179d9f2c47424` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:02.708333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4f3d98eef94428f9d8179d9f2c47424)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:02.710263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:02.757307 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4f3d98eef94428f9d8179d9f2c47424'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.768799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4f3d98eef94428f9d8179d9f2c47424'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.770701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:02.777060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4f3d98eef94428f9d8179d9f2c47424'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.785449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4f3d98eef94428f9d8179d9f2c47424'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.786212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-6fe24bb3-a4e7-4577-8734-7564761c2cc1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4f3d98eef94428f9d8179d9f2c47424'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:02.792146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1208/2419] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:02 2022] DELETE /identity/v3/users/d4f3d98eef94428f9d8179d9f2c47424 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:03.072327 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.074408 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:03.088263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7878c83-bb53-4ddc-ae8c-f40ad19ce638 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.147871 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.148530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.148814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.149206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] PATH_INFO: `/v3/projects/af9b7e8ea24e469d95ebc5beb801107d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.150759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af9b7e8ea24e469d95ebc5beb801107d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.152979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.174876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:03.205048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af9b7e8ea24e469d95ebc5beb801107d', 'request_id': 'req-fc71a27d-71a3-46df-bd15-6da534f023c1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:03.212134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af9b7e8ea24e469d95ebc5beb801107d', 'request_id': 'req-fc71a27d-71a3-46df-bd15-6da534f023c1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:03.217635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af9b7e8ea24e469d95ebc5beb801107d', 'request_id': 'req-fc71a27d-71a3-46df-bd15-6da534f023c1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:03.228758 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] The token cache is being invalidate because project af9b7e8ea24e469d95ebc5beb801107d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:03.229052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:03.229759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fc71a27d-71a3-46df-bd15-6da534f023c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:03.231459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1212/2420] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:03 2022] DELETE /identity/v3/projects/af9b7e8ea24e469d95ebc5beb801107d => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:03.236383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0e578-bdcd-4979-94a9-fb1ed5bafdc6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.236867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0e578-bdcd-4979-94a9-fb1ed5bafdc6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.236981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0e578-bdcd-4979-94a9-fb1ed5bafdc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.256852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-aac0e578-bdcd-4979-94a9-fb1ed5bafdc6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:03.258048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-aac0e578-bdcd-4979-94a9-fb1ed5bafdc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:03.322195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1209/2421] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:03.330373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242bd9b6-ba44-4d53-aacb-588084851a25 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.330676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242bd9b6-ba44-4d53-aacb-588084851a25 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.330969 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242bd9b6-ba44-4d53-aacb-588084851a25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.351197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-242bd9b6-ba44-4d53-aacb-588084851a25 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:03.352574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-242bd9b6-ba44-4d53-aacb-588084851a25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:03.365183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1213/2422] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:03.369466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0e578-bdcd-4979-94a9-fb1ed5bafdc6 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.379826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0e578-bdcd-4979-94a9-fb1ed5bafdc6 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.384499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.385113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.385396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.385643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.386396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:17:03.386700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.388352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.400313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1210/2423] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:17:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:17:03.404715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242bd9b6-ba44-4d53-aacb-588084851a25 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.414367 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242bd9b6-ba44-4d53-aacb-588084851a25 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.419695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.420308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.420584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.420844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.421506 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.423479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.454028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1214/2424] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:03 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:03.461451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.472974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f449341-c8bf-4cb3-b587-23598fdbf92c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.537882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.538511 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.538789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.539155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.539834 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.542011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.565748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1211/2425] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:03 2022] POST /identity/v3/users => generated 287 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:17:03.570865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.583665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be2339-faab-4791-ad1e-cae1d4452bf4 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.588951 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.589876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.590158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.590432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.591167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.592785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.604679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1215/2426] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:03 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:03.608787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.619152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad5b988-2c8f-4db0-a6fe-d67504773825 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.624783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592ae1ca-2a96-412c-a563-912c84b9947d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.625394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ae1ca-2a96-412c-a563-912c84b9947d admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.625673 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ae1ca-2a96-412c-a563-912c84b9947d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.625945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592ae1ca-2a96-412c-a563-912c84b9947d admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.626649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ae1ca-2a96-412c-a563-912c84b9947d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.628289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592ae1ca-2a96-412c-a563-912c84b9947d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.639482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1212/2427] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:17:03.643716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.654239 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab816fe-984d-4d49-a3fb-9327bf93b378 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.659745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.660409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.660699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.660948 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] PATH_INFO: `/v3/projects/8438d753dc6643dab71a423ffe57e8ca/users/6919ce499a6c4cf8a3aac9c0ee86bafa/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.677079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8438d753dc6643dab71a423ffe57e8ca, user_id=6919ce499a6c4cf8a3aac9c0ee86bafa, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.679773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.695334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1216/2428] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:03 2022] PUT /identity/v3/projects/8438d753dc6643dab71a423ffe57e8ca/users/6919ce499a6c4cf8a3aac9c0ee86bafa/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:03.702330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81963c29-20a4-4aea-8e5b-20420978b30b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.702620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81963c29-20a4-4aea-8e5b-20420978b30b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.702896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81963c29-20a4-4aea-8e5b-20420978b30b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.722823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-81963c29-20a4-4aea-8e5b-20420978b30b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:03.724100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-81963c29-20a4-4aea-8e5b-20420978b30b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:03.772387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:03.796868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1213/2429] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:03.806604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:03.808497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:03.869703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4a2e3b-541b-4c8a-a172-1990b8c25823 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:03.873713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:03.874342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:03.874627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:03.874905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:03.884141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:03.885590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:03.895071 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1217/2430] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:04.526164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188048b6-0e1f-467c-8a5c-517ec68a999d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:04.526742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188048b6-0e1f-467c-8a5c-517ec68a999d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:04.526742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188048b6-0e1f-467c-8a5c-517ec68a999d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:04.556532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-188048b6-0e1f-467c-8a5c-517ec68a999d None None] MFA Rules not processed for user `6919ce499a6c4cf8a3aac9c0ee86bafa`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:04.557825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-188048b6-0e1f-467c-8a5c-517ec68a999d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:04.631932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1214/2431] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:04 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:04.636739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:04.646872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcf7e53-cf68-4073-8ac9-bea9ceb1b36f service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:04.652246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:04.652874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:04.653153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:04.653423 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:04.654095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:04.656090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:04.694772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1218/2432] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:04 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:04.700268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188048b6-0e1f-467c-8a5c-517ec68a999d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:04.711975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188048b6-0e1f-467c-8a5c-517ec68a999d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:04.785767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:04.786599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:04.787009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:04.787442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:04.788532 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:04.791814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:04.821712 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1215/2433] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:04 2022] POST /identity/v3/users => generated 293 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:04.826339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:04.836934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1086f81f-fb2a-4158-b306-737c2a548289 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:04.841648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:04.842268 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:04.842548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:04.842860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:04.843585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:04.845215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:04.856031 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1219/2434] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:04.860278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:04.871310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862f3a63-ac5a-42cf-99b2-9615eea37219 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:04.876559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:04.877228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:04.877632 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:04.877928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] PATH_INFO: `/v3/projects/79876de535d844e5a57843bbc5b0e0a4/users/95280f3c7c984d7f915d7434e184bdbb/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:04.893378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79876de535d844e5a57843bbc5b0e0a4, user_id=95280f3c7c984d7f915d7434e184bdbb, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:04.896030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:04.912896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1216/2435] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:04 2022] PUT /identity/v3/projects/79876de535d844e5a57843bbc5b0e0a4/users/95280f3c7c984d7f915d7434e184bdbb/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:04.916918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:04.926995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb0ff5f-69db-4246-aed8-16fa74d7fa60 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:04.935955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:04.988185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca12434c-8af6-41b3-8c13-afe0c00a5e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:04.988793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca12434c-8af6-41b3-8c13-afe0c00a5e07 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:04.989077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca12434c-8af6-41b3-8c13-afe0c00a5e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:04.989373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca12434c-8af6-41b3-8c13-afe0c00a5e07 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:04.990065 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca12434c-8af6-41b3-8c13-afe0c00a5e07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:04.991742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca12434c-8af6-41b3-8c13-afe0c00a5e07 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:05.002290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1220/2436] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:04 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:17:05.006573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:05.017926 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b255b-4452-4446-974c-5f7cf4863418 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:05.023789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:05.024466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:05.024745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:05.025017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] PATH_INFO: `/v3/projects/79876de535d844e5a57843bbc5b0e0a4/users/95280f3c7c984d7f915d7434e184bdbb/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:05.027494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79876de535d844e5a57843bbc5b0e0a4, user_id=95280f3c7c984d7f915d7434e184bdbb, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:05.030283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:05.041441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1217/2437] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:05 2022] PUT /identity/v3/projects/79876de535d844e5a57843bbc5b0e0a4/users/95280f3c7c984d7f915d7434e184bdbb/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:05.722092 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c9dcb0-fde5-4977-9099-dcae523285e2 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:05.722684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c9dcb0-fde5-4977-9099-dcae523285e2 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:05.722684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c9dcb0-fde5-4977-9099-dcae523285e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:05.756965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-97c9dcb0-fde5-4977-9099-dcae523285e2 None None] MFA Rules not processed for user `95280f3c7c984d7f915d7434e184bdbb`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:05.759211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-97c9dcb0-fde5-4977-9099-dcae523285e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:05.841945 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1221/2438] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:05 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:05.853853 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:05.855674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:05.893078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:05.921686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab40798b-cf43-445c-aa64-db993f161cc8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:05.925585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:05.926392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:05.926688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:05.926990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:05.984695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:05.986133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:05.995309 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1218/2439] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:05 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:06.674128 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c9dcb0-fde5-4977-9099-dcae523285e2 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:06.728680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c9dcb0-fde5-4977-9099-dcae523285e2 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:06.732437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:06.733052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:06.733330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:06.733601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:06.735383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:06.783669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:06.799488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:06.801059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:06.811182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1222/2440] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:07.364825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:07.366569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:07.378330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0364743e-b0c9-4b63-a469-bc409e1ab230 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:07.383359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:07.384010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:07.384288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:07.384546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:07.393933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:07.395399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:07.404983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1219/2441] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:07 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:10.322597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:10.324308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:10.337456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3d5e6d-4ce3-48b0-9a34-75d096dd2e22 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:10.342191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:10.342851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:10.343167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:10.343445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] PATH_INFO: `/v3/users/fbdac72f58df4335b3790c5fb1609f5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:10.344639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbdac72f58df4335b3790c5fb1609f5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:10.346577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:10.405348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbdac72f58df4335b3790c5fb1609f5b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.412695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbdac72f58df4335b3790c5fb1609f5b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.421134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbdac72f58df4335b3790c5fb1609f5b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.433552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbdac72f58df4335b3790c5fb1609f5b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.434594 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0f67d91a-ac13-44ca-820c-d1a4e9463f91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbdac72f58df4335b3790c5fb1609f5b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.440919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1223/2442] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:10 2022] DELETE /identity/v3/users/fbdac72f58df4335b3790c5fb1609f5b => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:10.715889 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:10.734247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5007f6e5-bd1a-4ec9-9c61-88a07123f6d9 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:10.792442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:10.793115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:10.793398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:10.793676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] PATH_INFO: `/v3/projects/6daba0980548418ea98d4d0cf42ef92f` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:10.794845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] RBAC: Authorizing `identity:delete_project(project_id=6daba0980548418ea98d4d0cf42ef92f)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:10.796828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:10.843442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6daba0980548418ea98d4d0cf42ef92f', 'request_id': 'req-602b99bf-6647-45df-ab8c-00570f2b83df'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.851466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6daba0980548418ea98d4d0cf42ef92f', 'request_id': 'req-602b99bf-6647-45df-ab8c-00570f2b83df'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.857468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6daba0980548418ea98d4d0cf42ef92f', 'request_id': 'req-602b99bf-6647-45df-ab8c-00570f2b83df'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.869237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] The token cache is being invalidate because project 6daba0980548418ea98d4d0cf42ef92f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:10.869527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.870203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-602b99bf-6647-45df-ab8c-00570f2b83df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:10.871924 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1220/2443] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:10 2022] DELETE /identity/v3/projects/6daba0980548418ea98d4d0cf42ef92f => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:17:10.877570 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a705f9e-ab11-4433-a665-b731599d0c00 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:10.877933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a705f9e-ab11-4433-a665-b731599d0c00 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:10.878207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a705f9e-ab11-4433-a665-b731599d0c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:10.899081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8a705f9e-ab11-4433-a665-b731599d0c00 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:10.900282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8a705f9e-ab11-4433-a665-b731599d0c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:10.970393 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1224/2444] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:10.979064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc787a56-7c79-475c-a977-33cc9e59926d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:10.979373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc787a56-7c79-475c-a977-33cc9e59926d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:10.979671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc787a56-7c79-475c-a977-33cc9e59926d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:10.988837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:11.002838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-dc787a56-7c79-475c-a977-33cc9e59926d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:11.005215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dc787a56-7c79-475c-a977-33cc9e59926d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:11.015385 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1221/2445] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:11.019681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a705f9e-ab11-4433-a665-b731599d0c00 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.029765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a705f9e-ab11-4433-a665-b731599d0c00 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.034706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.035333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.035615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.035888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.036613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:17:11.036934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.038545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.050335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1225/2446] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:17:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:17:11.054830 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc787a56-7c79-475c-a977-33cc9e59926d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.064996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc787a56-7c79-475c-a977-33cc9e59926d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.070655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.071681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.071966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.072241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.072915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.074910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.107228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1222/2447] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:11 2022] POST /identity/v3/projects => generated 373 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:11.111497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.122574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e927e10-5c4f-4b2b-bf19-20e63dbfb707 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.166775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:11.188352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.189033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.189316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.189595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.190286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.192301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.217529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1226/2448] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:11 2022] POST /identity/v3/users => generated 287 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:11.222165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.233082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574d2983-422a-4743-9490-cb1865621950 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.238665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.239331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.239816 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.239922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.240838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.242159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.252710 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1223/2449] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:17:11.257073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.268226 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a92657a-a15e-4691-84c1-db3f36a68eda admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.273168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30577fed-e0bf-4614-b086-e6f84d5adf63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.273788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30577fed-e0bf-4614-b086-e6f84d5adf63 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.274080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30577fed-e0bf-4614-b086-e6f84d5adf63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.274357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30577fed-e0bf-4614-b086-e6f84d5adf63 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.275098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30577fed-e0bf-4614-b086-e6f84d5adf63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.276819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30577fed-e0bf-4614-b086-e6f84d5adf63 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.287827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1227/2450] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:11.292016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.302472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc8203e-8d53-416f-8c56-46df27485b26 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.308125 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.308791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.309070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.309348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] PATH_INFO: `/v3/projects/55313946954445649b97159fcde26a9e/users/46231c4eaa54470083424ac14dcff5a3/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.325009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55313946954445649b97159fcde26a9e, user_id=46231c4eaa54470083424ac14dcff5a3, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.327734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.342716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1224/2451] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:11 2022] PUT /identity/v3/projects/55313946954445649b97159fcde26a9e/users/46231c4eaa54470083424ac14dcff5a3/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:11.354499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96e7da7-70a1-45c7-a9c7-8e1631746cfb None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.355058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96e7da7-70a1-45c7-a9c7-8e1631746cfb None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.355277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96e7da7-70a1-45c7-a9c7-8e1631746cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.389131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c96e7da7-70a1-45c7-a9c7-8e1631746cfb None None] MFA Rules not processed for user `46231c4eaa54470083424ac14dcff5a3`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:11.390528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c96e7da7-70a1-45c7-a9c7-8e1631746cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:11.470043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1228/2452] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:11 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:11.474467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.542939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f53c570-e91b-46be-99ce-4b1456d6eff9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.550984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.551521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.551825 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.552106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.552792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.554991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.585748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1225/2453] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:11 2022] POST /identity/v3/projects => generated 373 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:11.590503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96e7da7-70a1-45c7-a9c7-8e1631746cfb None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.602505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96e7da7-70a1-45c7-a9c7-8e1631746cfb None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.687955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.688908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.689196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.689472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.690175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.692434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.721028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1229/2454] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:11 2022] POST /identity/v3/users => generated 293 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:17:11.723069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.735370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adefe59-9871-48e5-b7ba-ab8cb5edf98a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.741486 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.742208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.742560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.742895 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.743827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.745524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.758988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1226/2455] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:11 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:11.762752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.772981 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1306bf45-4b6e-4eb9-8590-a95d1790f997 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.777726 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.778474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.778811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.779124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] PATH_INFO: `/v3/projects/5251e09424fd469f893cf9d29dcf7cfb/users/c9d80a32a0cf4e0b99d7183bf06ce338/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.794774 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5251e09424fd469f893cf9d29dcf7cfb, user_id=c9d80a32a0cf4e0b99d7183bf06ce338, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.797495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.813592 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1230/2456] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:11 2022] PUT /identity/v3/projects/5251e09424fd469f893cf9d29dcf7cfb/users/c9d80a32a0cf4e0b99d7183bf06ce338/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:11.818432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.831200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad709acf-328f-4b37-a979-ebfc0898a464 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.888627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bf5a44-10a5-42f9-9be2-244e6fad7448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.889260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf5a44-10a5-42f9-9be2-244e6fad7448 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.889542 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf5a44-10a5-42f9-9be2-244e6fad7448 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.889818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf5a44-10a5-42f9-9be2-244e6fad7448 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.890519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bf5a44-10a5-42f9-9be2-244e6fad7448 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.892152 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bf5a44-10a5-42f9-9be2-244e6fad7448 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.902001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1227/2457] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:11 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:11.906317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:11.916767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bf6f6b-e64b-40d7-a3c0-ed6e137f03a3 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:11.922089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:11.922748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.923038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.923310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] PATH_INFO: `/v3/projects/5251e09424fd469f893cf9d29dcf7cfb/users/c9d80a32a0cf4e0b99d7183bf06ce338/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.925344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5251e09424fd469f893cf9d29dcf7cfb, user_id=c9d80a32a0cf4e0b99d7183bf06ce338, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:11.928009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:11.938850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1231/2458] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:11 2022] PUT /identity/v3/projects/5251e09424fd469f893cf9d29dcf7cfb/users/c9d80a32a0cf4e0b99d7183bf06ce338/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:17:11.950398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24db91f-8916-4d08-ba74-8c23d1bd1d63 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:11.950686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24db91f-8916-4d08-ba74-8c23d1bd1d63 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:11.950996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24db91f-8916-4d08-ba74-8c23d1bd1d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:11.972037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:11.980543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e24db91f-8916-4d08-ba74-8c23d1bd1d63 None None] MFA Rules not processed for user `c9d80a32a0cf4e0b99d7183bf06ce338`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:11.982024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e24db91f-8916-4d08-ba74-8c23d1bd1d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:12.056491 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1228/2459] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:11 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:17:12.068824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:12.070609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:12.078099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:12.134538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d76fc4-b907-47a9-b20b-fcf79a156e93 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:12.138431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:12.139108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:12.139399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:12.139669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:12.149575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:12.151917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:12.162049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1232/2460] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:12 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:12.252609 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24db91f-8916-4d08-ba74-8c23d1bd1d63 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:12.263368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24db91f-8916-4d08-ba74-8c23d1bd1d63 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:12.269099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3d8e4-73c3-44fe-bfc7-6daf6cf979bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:12.269249 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3d8e4-73c3-44fe-bfc7-6daf6cf979bd service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:12.269535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3d8e4-73c3-44fe-bfc7-6daf6cf979bd service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:12.269814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3d8e4-73c3-44fe-bfc7-6daf6cf979bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:12.338079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3d8e4-73c3-44fe-bfc7-6daf6cf979bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:12.339999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3d8e4-73c3-44fe-bfc7-6daf6cf979bd service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:12.350873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1229/2461] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:12 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:13.194235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:13.196700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:13.304693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfbeb3e-790e-4ed0-a902-10361f7d5cda service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:13.310684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:13.311502 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:13.311875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:13.312242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:13.314517 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:13.333223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:13.335051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:13.347373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1233/2462] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:13 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:22.346073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=57 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:23.342494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=58 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:24.022233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:24.022648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:24.022781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:24.043113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] MFA Rules not processed for user `c70c080fc8354ccf90391984baf6052f`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:24.044345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:24.108208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1230/2463] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:17:24.590389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:24.592147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:24.663987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49606ddf-5a27-46f0-a3d5-7508a4f74e5a service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:24.667912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:24.668532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:24.669050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:24.669050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:24.670978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:24.761261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:24.762722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:24.776017 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1234/2464] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:28.665671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:28.667556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:28.690796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83bdbbff-d419-4e60-b76e-7c336a4ce08f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:28.707558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:28.708481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:28.708955 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:28.709415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] PATH_INFO: `/v3/users/037c9f8229a44de198ebc3752f7a1966` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:28.711138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] RBAC: Authorizing `identity:delete_user(user_id=037c9f8229a44de198ebc3752f7a1966)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:28.713315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:28.779092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '037c9f8229a44de198ebc3752f7a1966'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:28.796200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '037c9f8229a44de198ebc3752f7a1966'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:28.804713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '037c9f8229a44de198ebc3752f7a1966'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:28.819604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '037c9f8229a44de198ebc3752f7a1966'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:28.820604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-375e7f82-7d84-40c5-8130-383fda1dd770 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '037c9f8229a44de198ebc3752f7a1966'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:28.835103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1231/2465] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:28 2022] DELETE /identity/v3/users/037c9f8229a44de198ebc3752f7a1966 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:29.024337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.035906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead60dae-dd7a-44a8-a89b-15d97bb0babb service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.097537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.098220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.098504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.098780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] PATH_INFO: `/v3/projects/ab741435041940cc9c7f02b27120bd90` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.099989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab741435041940cc9c7f02b27120bd90)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.101915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.148583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab741435041940cc9c7f02b27120bd90', 'request_id': 'req-21da5456-6544-4bab-ab0a-d377ebb8f40b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:29.151091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:29.157957 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab741435041940cc9c7f02b27120bd90', 'request_id': 'req-21da5456-6544-4bab-ab0a-d377ebb8f40b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:29.164795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab741435041940cc9c7f02b27120bd90', 'request_id': 'req-21da5456-6544-4bab-ab0a-d377ebb8f40b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:29.178338 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] The token cache is being invalidate because project ab741435041940cc9c7f02b27120bd90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:29.178634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:29.179342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:29.181182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1235/2466] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:29 2022] DELETE /identity/v3/projects/ab741435041940cc9c7f02b27120bd90 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:29.186311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa163d64-5e13-471a-af41-94320e697de5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.186604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa163d64-5e13-471a-af41-94320e697de5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.186884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa163d64-5e13-471a-af41-94320e697de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.201710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.203307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:29.208113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-aa163d64-5e13-471a-af41-94320e697de5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:29.209625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-aa163d64-5e13-471a-af41-94320e697de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:29.286768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1232/2467] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:29.293292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21da5456-6544-4bab-ab0a-d377ebb8f40b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.296043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e47969a-ed55-4fe8-a955-33fb4cea1adc None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.296684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e47969a-ed55-4fe8-a955-33fb4cea1adc None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.297287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e47969a-ed55-4fe8-a955-33fb4cea1adc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.299204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.299850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.300136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.300415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.302330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:29.315153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.316789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.328220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-8e47969a-ed55-4fe8-a955-33fb4cea1adc None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:29.329683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8e47969a-ed55-4fe8-a955-33fb4cea1adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:29.330495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1236/2468] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:29 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:29.339087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1233/2469] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:17:29.343166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.355550 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec9bf9c-f955-4788-8cc1-5f4f7d0745ff service placement] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.362156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.362858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.363171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.363450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.364209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:17:29.364560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.366880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.389626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1237/2470] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:17:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:17:29.391439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e47969a-ed55-4fe8-a955-33fb4cea1adc None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.402084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e47969a-ed55-4fe8-a955-33fb4cea1adc None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.407234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.407947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.408287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.409263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.410035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.415058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.431498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:29.465471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1234/2471] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:29 2022] POST /identity/v3/projects => generated 385 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:29.466673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.491581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5446673-c5d4-409d-953f-4852f68712c1 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.559635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.560358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.560700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.561130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.561875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.564264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.590799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1238/2472] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:29 2022] POST /identity/v3/users => generated 293 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:29.595069 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.605952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86d4eb6-c1fc-4c2c-a2bd-950d05fb6015 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.611976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.612617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.612900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.613174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.613888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.615594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.626448 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1235/2473] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:17:29.630405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.640816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466ec23d-2373-412e-b872-44d4249da768 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.645618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda4f6b9-0f2e-4dcf-a62f-01fd5636cb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.646373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda4f6b9-0f2e-4dcf-a62f-01fd5636cb7c admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.646756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda4f6b9-0f2e-4dcf-a62f-01fd5636cb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.647164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda4f6b9-0f2e-4dcf-a62f-01fd5636cb7c admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.648093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda4f6b9-0f2e-4dcf-a62f-01fd5636cb7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.649852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda4f6b9-0f2e-4dcf-a62f-01fd5636cb7c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.666810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.671724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1239/2474] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:29 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:29.677702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e99dff-b6bb-45a0-a45a-05c5ea7329fb admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.683146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.684122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.684503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.684878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] PATH_INFO: `/v3/projects/e6b0117c65a04365a08daf58e03470dc/users/740edf2471e14961bab709844ebd876d/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.701507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6b0117c65a04365a08daf58e03470dc, user_id=740edf2471e14961bab709844ebd876d, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.704899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.732893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1236/2475] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:29 2022] PUT /identity/v3/projects/e6b0117c65a04365a08daf58e03470dc/users/740edf2471e14961bab709844ebd876d/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:29.735743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.736033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.736339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.773643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] MFA Rules not processed for user `740edf2471e14961bab709844ebd876d`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:29.775314 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:29.802313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:29.843733 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.846100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:29.869086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1240/2476] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:29.882571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:29.884075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:29.922436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a331d2-b3ae-413c-a3ca-0e72ce8bfdfd admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.928170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.928961 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.929339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.929714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] PATH_INFO: `/v3/users/6919ce499a6c4cf8a3aac9c0ee86bafa` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:29.931311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6919ce499a6c4cf8a3aac9c0ee86bafa)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:29.933714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:29.983682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e56bc65-bf85-432f-a7a8-4eb30e1eebb5 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:29.989561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:29.990171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:29.990486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:29.990765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:30.004530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6919ce499a6c4cf8a3aac9c0ee86bafa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.004848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:30.007048 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:30.020521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6919ce499a6c4cf8a3aac9c0ee86bafa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.022823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1241/2477] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:30.028753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6919ce499a6c4cf8a3aac9c0ee86bafa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.039596 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6919ce499a6c4cf8a3aac9c0ee86bafa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.040397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6919ce499a6c4cf8a3aac9c0ee86bafa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.048663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1237/2478] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:29 2022] DELETE /identity/v3/users/6919ce499a6c4cf8a3aac9c0ee86bafa => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:30.249961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:30.265743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7e3253-bd31-42b2-86b0-afd13833a526 service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:30.353518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:30.354287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:30.354575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:30.354848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] PATH_INFO: `/v3/projects/8438d753dc6643dab71a423ffe57e8ca` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:30.356148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] RBAC: Authorizing `identity:delete_project(project_id=8438d753dc6643dab71a423ffe57e8ca)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:30.359390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:30.424688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8438d753dc6643dab71a423ffe57e8ca', 'request_id': 'req-28338f58-2e8b-4e84-9406-99fd7d6b736a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.427857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:30.433588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8438d753dc6643dab71a423ffe57e8ca', 'request_id': 'req-28338f58-2e8b-4e84-9406-99fd7d6b736a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.442693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8438d753dc6643dab71a423ffe57e8ca', 'request_id': 'req-28338f58-2e8b-4e84-9406-99fd7d6b736a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.459735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] The token cache is being invalidate because project 8438d753dc6643dab71a423ffe57e8ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:30.460206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.460963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.462901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1242/2479] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:30 2022] DELETE /identity/v3/projects/8438d753dc6643dab71a423ffe57e8ca => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:30.467237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:30.469378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:30.484800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1982b408-24b0-4497-b9ab-1756fceb9078 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:30.520184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:30.548535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:30.549189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:30.549472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:30.549751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] PATH_INFO: `/v3/users/95280f3c7c984d7f915d7434e184bdbb` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:30.550912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] RBAC: Authorizing `identity:delete_user(user_id=95280f3c7c984d7f915d7434e184bdbb)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:30.552876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:30.599374 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95280f3c7c984d7f915d7434e184bdbb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.611404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95280f3c7c984d7f915d7434e184bdbb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.618302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95280f3c7c984d7f915d7434e184bdbb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.627263 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95280f3c7c984d7f915d7434e184bdbb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.627960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-cf254af0-a7bc-47c2-a0fe-031e6d06ab5c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95280f3c7c984d7f915d7434e184bdbb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.634288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1238/2480] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:30 2022] DELETE /identity/v3/users/95280f3c7c984d7f915d7434e184bdbb => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:30.821989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:30.823678 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:30.837385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28338f58-2e8b-4e84-9406-99fd7d6b736a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:30.900930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:30.901605 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:30.901889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:30.902172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] PATH_INFO: `/v3/projects/79876de535d844e5a57843bbc5b0e0a4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:30.903481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=79876de535d844e5a57843bbc5b0e0a4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:30.905513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:30.959984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79876de535d844e5a57843bbc5b0e0a4', 'request_id': 'req-eb4d80cf-6683-4661-8178-1444bdb6c8ee'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.968590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79876de535d844e5a57843bbc5b0e0a4', 'request_id': 'req-eb4d80cf-6683-4661-8178-1444bdb6c8ee'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.976293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79876de535d844e5a57843bbc5b0e0a4', 'request_id': 'req-eb4d80cf-6683-4661-8178-1444bdb6c8ee'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.989014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] The token cache is being invalidate because project 79876de535d844e5a57843bbc5b0e0a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:30.989258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.990003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-eb4d80cf-6683-4661-8178-1444bdb6c8ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:30.996343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1243/2481] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:30 2022] DELETE /identity/v3/projects/79876de535d844e5a57843bbc5b0e0a4 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:30.998318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1134787d-4adb-4143-99de-599377a15326 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:30.999372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1134787d-4adb-4143-99de-599377a15326 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:30.999764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1134787d-4adb-4143-99de-599377a15326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.021947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1134787d-4adb-4143-99de-599377a15326 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:31.023402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1134787d-4adb-4143-99de-599377a15326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:31.095548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1239/2482] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:31.097610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda0194c-5113-4b2b-9140-f7170dad7c2a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.097972 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda0194c-5113-4b2b-9140-f7170dad7c2a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.098324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda0194c-5113-4b2b-9140-f7170dad7c2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.118734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-eda0194c-5113-4b2b-9140-f7170dad7c2a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:31.119987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-eda0194c-5113-4b2b-9140-f7170dad7c2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:31.128701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1244/2483] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:31.133327 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1134787d-4adb-4143-99de-599377a15326 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.143670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1134787d-4adb-4143-99de-599377a15326 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.148520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.149165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.149449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.149728 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.150442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:17:31.150754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.152418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.167008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1240/2484] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:17:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:17:31.169589 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda0194c-5113-4b2b-9140-f7170dad7c2a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.179509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda0194c-5113-4b2b-9140-f7170dad7c2a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.184766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.185471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.185813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.186159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.186904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.188961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.222405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1245/2485] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:31 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:31.223554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.233724 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4df6fc8-60e3-4f84-8c35-d1261b4f879c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.273339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:31.292990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.293613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.293894 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.294171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.294839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.296903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.318973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1241/2486] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:31 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:31.323579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.334280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6520e693-10f1-4646-b899-f92af2bed866 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.339007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.339639 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.339916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.340185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.340886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.342504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.352480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1246/2487] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:31.356294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.366909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adf7c43-cd4c-4473-91e9-783ec509329d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.372232 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f582eac-36ba-4e54-b88f-fd19f70ba78a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.372845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f582eac-36ba-4e54-b88f-fd19f70ba78a admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.373124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f582eac-36ba-4e54-b88f-fd19f70ba78a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.373400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f582eac-36ba-4e54-b88f-fd19f70ba78a admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.374084 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f582eac-36ba-4e54-b88f-fd19f70ba78a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.375748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f582eac-36ba-4e54-b88f-fd19f70ba78a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.387377 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1242/2488] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:31.391672 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.401700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28854820-422f-4afa-891e-03358241eda6 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.406413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.407116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.407396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.407670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] PATH_INFO: `/v3/projects/f7acc7f1eac74b388df728b46fe171f9/users/429b2dbd81b5481784365775e5f4c59c/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.422588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7acc7f1eac74b388df728b46fe171f9, user_id=429b2dbd81b5481784365775e5f4c59c, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.425443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.440575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1247/2489] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:31 2022] PUT /identity/v3/projects/f7acc7f1eac74b388df728b46fe171f9/users/429b2dbd81b5481784365775e5f4c59c/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:31.452684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69b09d7-7bfa-46c2-830a-d745762d08c2 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.452968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69b09d7-7bfa-46c2-830a-d745762d08c2 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.453240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69b09d7-7bfa-46c2-830a-d745762d08c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.482256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f69b09d7-7bfa-46c2-830a-d745762d08c2 None None] MFA Rules not processed for user `429b2dbd81b5481784365775e5f4c59c`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:31.483530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f69b09d7-7bfa-46c2-830a-d745762d08c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:31.559862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1243/2490] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:31.564199 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.582554 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:31.643147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69d49f7-9658-4146-aa50-0091986d31d8 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.648285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.648919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.649218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.649481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.650156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.652119 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.680583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1248/2491] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:31 2022] POST /identity/v3/projects => generated 385 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:31.684763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69b09d7-7bfa-46c2-830a-d745762d08c2 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.697566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69b09d7-7bfa-46c2-830a-d745762d08c2 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.769839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.770483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.770769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.771082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.772248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.774212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.806255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1244/2492] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:31 2022] POST /identity/v3/users => generated 299 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:31.810417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.828608 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f55d6-1bd8-4871-a887-4547d5c7be30 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.841254 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.841933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.842183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.842486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.843352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.846287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.861499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1249/2493] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:31 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:31.866177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.878115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b916668-23c6-43ad-9bd9-263acd71b9d1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:31.884421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:31.885741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:31.886166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:31.886601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] PATH_INFO: `/v3/projects/120fbee309e2400ebbfca479a9a5f015/users/2785c3fb0b8940738d5c30683b6d5e84/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:31.911843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=120fbee309e2400ebbfca479a9a5f015, user_id=2785c3fb0b8940738d5c30683b6d5e84, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:31.914612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:31.928506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1245/2494] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:31 2022] PUT /identity/v3/projects/120fbee309e2400ebbfca479a9a5f015/users/2785c3fb0b8940738d5c30683b6d5e84/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:31.932595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:31.943718 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2d2692-4873-483d-9685-135c6b594b13 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:32.003256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d53222-8428-4cf0-8b9d-ce4621f08c3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:32.003889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d53222-8428-4cf0-8b9d-ce4621f08c3d admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:32.004167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d53222-8428-4cf0-8b9d-ce4621f08c3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:32.004437 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d53222-8428-4cf0-8b9d-ce4621f08c3d admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:32.005136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d53222-8428-4cf0-8b9d-ce4621f08c3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:32.006823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d53222-8428-4cf0-8b9d-ce4621f08c3d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:32.017416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1250/2495] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:31 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:32.021857 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:32.033213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba43a069-77dc-4ca0-a5a4-b2f2eb0827e2 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:32.038707 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:32.039416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:32.039700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:32.039979 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] PATH_INFO: `/v3/projects/120fbee309e2400ebbfca479a9a5f015/users/2785c3fb0b8940738d5c30683b6d5e84/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:32.041976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] RBAC: Authorizing `identity:create_grant(project_id=120fbee309e2400ebbfca479a9a5f015, user_id=2785c3fb0b8940738d5c30683b6d5e84, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:32.044681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:32.051556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:32.056536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1246/2496] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:32 2022] PUT /identity/v3/projects/120fbee309e2400ebbfca479a9a5f015/users/2785c3fb0b8940738d5c30683b6d5e84/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:32.066697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:32.066997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:32.067294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:32.095745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] MFA Rules not processed for user `2785c3fb0b8940738d5c30683b6d5e84`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:32.096963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:32.165551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1251/2497] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:32 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:32.177523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:32.179531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:32.238472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd916f-444f-4cae-8b14-17746bda1ecc admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:32.242827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fdda66-55c0-4b4e-b9a0-bda05dba9065 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:32.243474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fdda66-55c0-4b4e-b9a0-bda05dba9065 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:32.243752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fdda66-55c0-4b4e-b9a0-bda05dba9065 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:32.244011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fdda66-55c0-4b4e-b9a0-bda05dba9065 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:32.305072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fdda66-55c0-4b4e-b9a0-bda05dba9065 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:32.306665 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fdda66-55c0-4b4e-b9a0-bda05dba9065 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:32.316386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1247/2498] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:37.572018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:37.572471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:37.577928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:37.647822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434c4062-bc37-4f5c-a819-f11a29d18616 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:37.652406 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:37.653060 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:37.653341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:37.653876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] PATH_INFO: `/v3/users/740edf2471e14961bab709844ebd876d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:37.655084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] RBAC: Authorizing `identity:delete_user(user_id=740edf2471e14961bab709844ebd876d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:37.657033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:37.702690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '740edf2471e14961bab709844ebd876d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:37.709609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '740edf2471e14961bab709844ebd876d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:37.718218 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '740edf2471e14961bab709844ebd876d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:37.729993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '740edf2471e14961bab709844ebd876d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:37.730655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '740edf2471e14961bab709844ebd876d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:37.736627 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1252/2499] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:37 2022] DELETE /identity/v3/users/740edf2471e14961bab709844ebd876d => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:37.741386 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b87414-a2f3-49ac-b267-120504945154 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:37.741867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b87414-a2f3-49ac-b267-120504945154 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:37.741962 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b87414-a2f3-49ac-b267-120504945154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:37.761939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-09b87414-a2f3-49ac-b267-120504945154 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:37.763202 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-09b87414-a2f3-49ac-b267-120504945154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:37.833434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1248/2500] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:17:37.840352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:37.841946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:37.902184 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06491e6-1f78-4a16-945a-6cd53413ba1c admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:37.906811 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12ad455-dbbb-423f-825a-abdaecb5dda7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:37.907474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ad455-dbbb-423f-825a-abdaecb5dda7 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:37.907760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ad455-dbbb-423f-825a-abdaecb5dda7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:37.908057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12ad455-dbbb-423f-825a-abdaecb5dda7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:37.918793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ad455-dbbb-423f-825a-abdaecb5dda7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:37.920427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12ad455-dbbb-423f-825a-abdaecb5dda7 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:37.931383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1253/2501] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:38.245495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b87414-a2f3-49ac-b267-120504945154 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.247024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-09b87414-a2f3-49ac-b267-120504945154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:38.259919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b87414-a2f3-49ac-b267-120504945154 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.266104 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.266835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.267134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.267338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] PATH_INFO: `/v3/projects/e6b0117c65a04365a08daf58e03470dc` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.269404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6b0117c65a04365a08daf58e03470dc)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.271457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.326813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6b0117c65a04365a08daf58e03470dc', 'request_id': 'req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:38.328496 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:38.337185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6b0117c65a04365a08daf58e03470dc', 'request_id': 'req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:38.343015 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6b0117c65a04365a08daf58e03470dc', 'request_id': 'req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:38.354212 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] The token cache is being invalidate because project e6b0117c65a04365a08daf58e03470dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:38.354502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:38.355158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c050b304-e5a1-4a13-9c5a-e049a55c7a4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:38.356701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1249/2502] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:38 2022] DELETE /identity/v3/projects/e6b0117c65a04365a08daf58e03470dc => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:38.361878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e535629-08e9-4d48-8b46-52ab0511e9f6 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.362171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e535629-08e9-4d48-8b46-52ab0511e9f6 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.362453 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e535629-08e9-4d48-8b46-52ab0511e9f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.381918 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7e535629-08e9-4d48-8b46-52ab0511e9f6 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:38.383193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7e535629-08e9-4d48-8b46-52ab0511e9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:38.401940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:38.445162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1254/2503] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:38.453661 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c3adbf-26bb-4fa2-ae5c-06b5731c7ef2 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.453945 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c3adbf-26bb-4fa2-ae5c-06b5731c7ef2 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.454220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c3adbf-26bb-4fa2-ae5c-06b5731c7ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.475880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c1c3adbf-26bb-4fa2-ae5c-06b5731c7ef2 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:38.477083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c1c3adbf-26bb-4fa2-ae5c-06b5731c7ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:38.486550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1250/2504] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:38.491480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e535629-08e9-4d48-8b46-52ab0511e9f6 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.506514 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e535629-08e9-4d48-8b46-52ab0511e9f6 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.513526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.514149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.514427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.514706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.515414 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:17:38.515729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.518151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.530680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1255/2505] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:17:38.535113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c3adbf-26bb-4fa2-ae5c-06b5731c7ef2 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.544493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c3adbf-26bb-4fa2-ae5c-06b5731c7ef2 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.549367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.549995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.550265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.550535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.551283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.553176 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.581208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1251/2506] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:38 2022] POST /identity/v3/projects => generated 355 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:38.585355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.596012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e575c6-3c16-4e5f-a3dc-f5f9b2ced82d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.668501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.669142 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.669427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.670742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.684844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.684844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.705347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1256/2507] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:38 2022] POST /identity/v3/users => generated 278 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:17:38.709583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.721246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4950be-c81e-4e62-9ace-7c119142842c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.727055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.727722 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.728002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.728663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.728969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.730568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.743159 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1252/2508] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:38.747116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.748553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.750186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:38.758194 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2802d79-bc63-404b-9414-b37b5a9e352b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.763848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.764465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.764749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.765020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.765725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.767431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.780799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1257/2509] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:38 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:38.785547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.795511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563b9b5-b789-4686-a6c0-70c76820253a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.800099 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdefcd3-2b55-4631-96a0-5d6da3ccd99f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.800779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdefcd3-2b55-4631-96a0-5d6da3ccd99f admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.801054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdefcd3-2b55-4631-96a0-5d6da3ccd99f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.801330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdefcd3-2b55-4631-96a0-5d6da3ccd99f admin admin] PATH_INFO: `/v3/projects/2309169db85e4ad19a5ba252fa776c8d/users/73178b9015154fe2b78c72bc1ebe6ed8/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.810778 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbab68ce-45a7-45c2-bda1-6bfecbe02ea0 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:38.814495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:38.815155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.815441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.815719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.816080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdefcd3-2b55-4631-96a0-5d6da3ccd99f admin admin] RBAC: Authorizing `identity:create_grant(project_id=2309169db85e4ad19a5ba252fa776c8d, user_id=73178b9015154fe2b78c72bc1ebe6ed8, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.817537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:38.818779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdefcd3-2b55-4631-96a0-5d6da3ccd99f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.835089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1258/2510] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:38 2022] PUT /identity/v3/projects/2309169db85e4ad19a5ba252fa776c8d/users/73178b9015154fe2b78c72bc1ebe6ed8/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:38.846714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55346912-9094-4cc4-932d-ed214ffc6f76 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:38.847028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55346912-9094-4cc4-932d-ed214ffc6f76 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:38.847310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55346912-9094-4cc4-932d-ed214ffc6f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:38.875697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-55346912-9094-4cc4-932d-ed214ffc6f76 None None] MFA Rules not processed for user `73178b9015154fe2b78c72bc1ebe6ed8`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:38.877039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-55346912-9094-4cc4-932d-ed214ffc6f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:38.882408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:38.883909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:38.893575 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1253/2511] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:38 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:17:38.934024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:38.947098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1259/2512] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:38 2022] POST /identity/v3/auth/tokens => generated 3222 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:17:38.949032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:38.960043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec963eb4-1e29-4999-b57a-e28bc5adb397 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.009718 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:INSERT=2,keystone:DELETE=3,keystone:SELECT=95 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:39.029369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.030005 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.030287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.030563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.031289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.033319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.064462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1254/2513] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:38 2022] POST /identity/v3/projects => generated 357 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:39.065504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55346912-9094-4cc4-932d-ed214ffc6f76 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:39.077019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55346912-9094-4cc4-932d-ed214ffc6f76 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.161691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.162427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.162854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.163249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.164012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.166108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.193233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1260/2514] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:39 2022] POST /identity/v3/users => generated 285 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:39.197696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:39.209008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a47a064-86da-41cb-bdfb-2daf812c04e1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.214009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.214640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.214921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.215229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.216068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.217689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.229401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1255/2515] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:39 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:39.233904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:39.244335 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486891cc-258b-4b91-b5c5-002510602cec admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.248967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.249751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.250090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.250419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] PATH_INFO: `/v3/projects/e2f4f8a46fa746119cfde3da4213bde9/users/3731dfa871da4831bfa34b43b83e833e/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.266049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2f4f8a46fa746119cfde3da4213bde9, user_id=3731dfa871da4831bfa34b43b83e833e, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.269496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.283347 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1261/2516] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:39 2022] PUT /identity/v3/projects/e2f4f8a46fa746119cfde3da4213bde9/users/3731dfa871da4831bfa34b43b83e833e/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:39.287563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:39.297752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eed30eb-a099-4083-a76f-62b96a037480 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.359243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b519829e-f4f8-4119-80ce-8c2113f08e10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.359882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519829e-f4f8-4119-80ce-8c2113f08e10 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.361843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519829e-f4f8-4119-80ce-8c2113f08e10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.361843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519829e-f4f8-4119-80ce-8c2113f08e10 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.361843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b519829e-f4f8-4119-80ce-8c2113f08e10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.362998 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b519829e-f4f8-4119-80ce-8c2113f08e10 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.378123 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1256/2517] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:39 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:39.382364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:39.393412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcfd9ce-418d-48db-8a02-163c85e9d2b0 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.398266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.399136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.399415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.399690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] PATH_INFO: `/v3/projects/e2f4f8a46fa746119cfde3da4213bde9/users/3731dfa871da4831bfa34b43b83e833e/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.401615 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2f4f8a46fa746119cfde3da4213bde9, user_id=3731dfa871da4831bfa34b43b83e833e, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.404306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.414739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1262/2518] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:39 2022] PUT /identity/v3/projects/e2f4f8a46fa746119cfde3da4213bde9/users/3731dfa871da4831bfa34b43b83e833e/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:39.425266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3569e122-5a52-4bba-af6a-3e0df2284b11 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.425551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3569e122-5a52-4bba-af6a-3e0df2284b11 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.425824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3569e122-5a52-4bba-af6a-3e0df2284b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.455197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3569e122-5a52-4bba-af6a-3e0df2284b11 None None] MFA Rules not processed for user `3731dfa871da4831bfa34b43b83e833e`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:39.456409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3569e122-5a52-4bba-af6a-3e0df2284b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:39.544430 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1257/2519] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:39 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:17:39.556362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:39.617858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3a73f6-9541-42d5-aaba-f547e2a8090e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.621818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.622493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.622791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.623111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.632942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.634396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.644869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1263/2520] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:39 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:39.766999 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3569e122-5a52-4bba-af6a-3e0df2284b11 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:39.776789 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3569e122-5a52-4bba-af6a-3e0df2284b11 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:39.780595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac0069e-20ec-4038-9559-eb30fc32e3d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:39.781214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0069e-20ec-4038-9559-eb30fc32e3d4 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:39.781490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0069e-20ec-4038-9559-eb30fc32e3d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:39.781763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0069e-20ec-4038-9559-eb30fc32e3d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:39.843900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:39.862359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac0069e-20ec-4038-9559-eb30fc32e3d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:39.863835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac0069e-20ec-4038-9559-eb30fc32e3d4 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:39.872985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1258/2521] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:39 2022] GET /identity/v3/auth/tokens => generated 3222 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:49.156075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:49.157720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:49.214924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:49.232621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd590d10-72ba-451f-b84c-d41e5956da8f service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:49.237845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:49.238531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:49.238801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:49.239102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] PATH_INFO: `/v3/users/429b2dbd81b5481784365775e5f4c59c` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:49.240359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] RBAC: Authorizing `identity:delete_user(user_id=429b2dbd81b5481784365775e5f4c59c)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:49.242416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:49.297539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '429b2dbd81b5481784365775e5f4c59c'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.304412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '429b2dbd81b5481784365775e5f4c59c'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.313154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '429b2dbd81b5481784365775e5f4c59c'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.326271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '429b2dbd81b5481784365775e5f4c59c'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.326914 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '429b2dbd81b5481784365775e5f4c59c'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.333022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1264/2522] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:49 2022] DELETE /identity/v3/users/429b2dbd81b5481784365775e5f4c59c => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:49.337726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:49.338254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:49.338405 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:49.358484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:49.359762 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:49.418655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1259/2523] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:49.428617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:49.430342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:49.498033 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4340a7-9f39-428b-9387-5751fb2a472b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:49.502364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:49.503024 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:49.503305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:49.503580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:49.513046 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:49.514456 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:49.524370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1265/2524] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:49.805406 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:49.807635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:49.822011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4461b1dc-22a0-4692-9635-52c697d57226 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:49.827193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:49.828063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:49.828354 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:49.828615 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] PATH_INFO: `/v3/projects/f7acc7f1eac74b388df728b46fe171f9` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:49.830076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7acc7f1eac74b388df728b46fe171f9)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:49.832081 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:49.882395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7acc7f1eac74b388df728b46fe171f9', 'request_id': 'req-607b9a9e-da69-4650-b7f6-f0b969ace842'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.892656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7acc7f1eac74b388df728b46fe171f9', 'request_id': 'req-607b9a9e-da69-4650-b7f6-f0b969ace842'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.898978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7acc7f1eac74b388df728b46fe171f9', 'request_id': 'req-607b9a9e-da69-4650-b7f6-f0b969ace842'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.911048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] The token cache is being invalidate because project f7acc7f1eac74b388df728b46fe171f9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:49.911338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.911977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:49.913520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1260/2525] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:49 2022] DELETE /identity/v3/projects/f7acc7f1eac74b388df728b46fe171f9 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:49.917548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:49.927626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d5ff5-b2dd-406f-a395-6ad93df3aa82 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:49.997384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:49.998284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:49.998721 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:49.999204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] PATH_INFO: `/v3/users/2785c3fb0b8940738d5c30683b6d5e84` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.000826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2785c3fb0b8940738d5c30683b6d5e84)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:50.003124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:50.023135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:50.069381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2785c3fb0b8940738d5c30683b6d5e84'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.081752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2785c3fb0b8940738d5c30683b6d5e84'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.092877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2785c3fb0b8940738d5c30683b6d5e84'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.109926 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2785c3fb0b8940738d5c30683b6d5e84'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.111051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f32afdf3-191f-4d30-b84f-2b35139a0b08 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2785c3fb0b8940738d5c30683b6d5e84'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.119908 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1266/2526] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:17:49 2022] DELETE /identity/v3/users/2785c3fb0b8940738d5c30683b6d5e84 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:17:50.427497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:50.429194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:50.442779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607b9a9e-da69-4650-b7f6-f0b969ace842 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:50.500626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:50.501217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.501520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.501793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] PATH_INFO: `/v3/projects/120fbee309e2400ebbfca479a9a5f015` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.503086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] RBAC: Authorizing `identity:delete_project(project_id=120fbee309e2400ebbfca479a9a5f015)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:50.505028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:50.513605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:50.555191 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '120fbee309e2400ebbfca479a9a5f015', 'request_id': 'req-c548fac2-2051-4d3b-b711-2db517ba9678'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.564329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '120fbee309e2400ebbfca479a9a5f015', 'request_id': 'req-c548fac2-2051-4d3b-b711-2db517ba9678'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.569939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '120fbee309e2400ebbfca479a9a5f015', 'request_id': 'req-c548fac2-2051-4d3b-b711-2db517ba9678'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.581342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] The token cache is being invalidate because project 120fbee309e2400ebbfca479a9a5f015 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:17:50.581629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.582269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c548fac2-2051-4d3b-b711-2db517ba9678 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:17:50.583837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1261/2527] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:17:50 2022] DELETE /identity/v3/projects/120fbee309e2400ebbfca479a9a5f015 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:17:50.590265 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82f9230-c50a-449e-924a-8bae357783bc None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.590759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82f9230-c50a-449e-924a-8bae357783bc None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.590864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82f9230-c50a-449e-924a-8bae357783bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.615848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a82f9230-c50a-449e-924a-8bae357783bc None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:50.617221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a82f9230-c50a-449e-924a-8bae357783bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:50.682814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1267/2528] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:17:50.692108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b5c6ae-e202-4b44-ac93-ab2f55c983b3 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.692326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b5c6ae-e202-4b44-ac93-ab2f55c983b3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.692605 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b5c6ae-e202-4b44-ac93-ab2f55c983b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.714805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c9b5c6ae-e202-4b44-ac93-ab2f55c983b3 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:50.716067 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c9b5c6ae-e202-4b44-ac93-ab2f55c983b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:50.724597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1262/2529] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:50.729326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82f9230-c50a-449e-924a-8bae357783bc None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:50.740001 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82f9230-c50a-449e-924a-8bae357783bc None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:50.745645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:50.746262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.746537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.746807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.747556 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:17:50.747863 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:50.749518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:50.761755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1268/2530] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:17:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:17:50.766368 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b5c6ae-e202-4b44-ac93-ab2f55c983b3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:50.776598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b5c6ae-e202-4b44-ac93-ab2f55c983b3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:50.783187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:50.783910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.784185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.784452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.785120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:50.787042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:50.816454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1263/2531] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:17:50 2022] POST /identity/v3/projects => generated 371 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:50.821280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:50.832651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2096ae-6049-4e48-9727-ddecbfb8a94a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:50.893255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:50.893873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.894147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.894418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.895109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:50.897041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:50.919146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1269/2532] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:17:50 2022] POST /identity/v3/users => generated 286 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:17:50.923433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:50.934422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a3c15-59fa-4512-a189-57a404626f1f admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:50.940090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:50.940741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.941040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.941318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.942023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:50.943746 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:50.954289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1264/2533] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:50.958012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:50.968489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7955b2-dffa-40ea-954e-d24abe76224e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:50.973587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0440b6ab-74ca-4483-95d4-552e1a5a1483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:50.974200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0440b6ab-74ca-4483-95d4-552e1a5a1483 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:50.974479 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0440b6ab-74ca-4483-95d4-552e1a5a1483 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:50.974748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0440b6ab-74ca-4483-95d4-552e1a5a1483 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:50.975573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0440b6ab-74ca-4483-95d4-552e1a5a1483 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:50.977185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0440b6ab-74ca-4483-95d4-552e1a5a1483 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:50.979242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:50.991243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1270/2534] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:17:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:17:50.995880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:51.007383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6ea94b-8cd0-415d-8793-5ed28d4d1fbc admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:51.012626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:51.013373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:51.013681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:51.013964 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] PATH_INFO: `/v3/projects/ad3b89efcbc94f19a867fe700046b0f2/users/72614ab651224f4b9c0f2fb90806aadf/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:51.033352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad3b89efcbc94f19a867fe700046b0f2, user_id=72614ab651224f4b9c0f2fb90806aadf, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:51.037656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:51.053694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1265/2535] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:17:50 2022] PUT /identity/v3/projects/ad3b89efcbc94f19a867fe700046b0f2/users/72614ab651224f4b9c0f2fb90806aadf/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:17:51.059508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f13533e-9f6a-4838-bc28-dab4896eacc3 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:51.059870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f13533e-9f6a-4838-bc28-dab4896eacc3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:51.060209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f13533e-9f6a-4838-bc28-dab4896eacc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:51.083753 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-0f13533e-9f6a-4838-bc28-dab4896eacc3 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:51.085000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0f13533e-9f6a-4838-bc28-dab4896eacc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:51.146191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1271/2536] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:51.156376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:51.158237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:51.217265 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731a115c-5967-4fe3-bde0-33cac521886e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:51.221181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:51.221818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:51.222108 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:51.222378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:51.232104 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:51.233536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:51.243210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1266/2537] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:17:52.027089 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919716ff-e0d9-4ca7-ac64-0e24bdf6c851 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:52.027652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919716ff-e0d9-4ca7-ac64-0e24bdf6c851 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:52.027652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919716ff-e0d9-4ca7-ac64-0e24bdf6c851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:52.057836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-919716ff-e0d9-4ca7-ac64-0e24bdf6c851 None None] MFA Rules not processed for user `72614ab651224f4b9c0f2fb90806aadf`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:52.059820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-919716ff-e0d9-4ca7-ac64-0e24bdf6c851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:52.126814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1272/2538] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:52 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:17:52.138246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:17:52.139731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:52.181132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:17:52.200407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a2b3a-c782-4dd5-adb3-99e58d604a0f service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:17:52.204061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:17:52.204690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:52.204977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:52.205256 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:52.213896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:17:52.215415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:17:52.224798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1267/2539] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:17:52 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:17:53.607833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:17:53.608321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:17:53.608512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:17:53.627754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] MFA Rules not processed for user `72614ab651224f4b9c0f2fb90806aadf`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:17:53.628962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:17:53.639485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1273/2540] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:17:53 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:01.083105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:01.094351 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0100515c-8c20-4e14-ae3f-c9377203bf38 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:01.100908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29503aa-7e69-4de7-962a-c172fafdcc20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:01.101680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29503aa-7e69-4de7-962a-c172fafdcc20 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:01.102034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29503aa-7e69-4de7-962a-c172fafdcc20 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:01.102318 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29503aa-7e69-4de7-962a-c172fafdcc20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:01.105651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e29503aa-7e69-4de7-962a-c172fafdcc20 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:01.184339 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29503aa-7e69-4de7-962a-c172fafdcc20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:01.186040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29503aa-7e69-4de7-962a-c172fafdcc20 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:01.197062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1268/2541] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:01 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:02.900444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:02.902008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:02.917390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79da37d8-dd1b-4b1f-81bc-e62440839649 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:02.922416 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:02.923188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:02.923553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:02.923892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] PATH_INFO: `/v3/users/73178b9015154fe2b78c72bc1ebe6ed8` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:02.925154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] RBAC: Authorizing `identity:delete_user(user_id=73178b9015154fe2b78c72bc1ebe6ed8)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:02.927225 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:02.982203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73178b9015154fe2b78c72bc1ebe6ed8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:02.994534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73178b9015154fe2b78c72bc1ebe6ed8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.005958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73178b9015154fe2b78c72bc1ebe6ed8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.024982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:03.028801 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73178b9015154fe2b78c72bc1ebe6ed8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.030039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73178b9015154fe2b78c72bc1ebe6ed8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.039966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1274/2542] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:02 2022] DELETE /identity/v3/users/73178b9015154fe2b78c72bc1ebe6ed8 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:03.044472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:03.044975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:03.045264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:03.066183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:03.067432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:03.132136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1269/2543] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:03.142135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:03.143607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:03.203076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5bc9c-19ae-4a2b-aee4-5f57ccd5c22b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:03.206682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:03.207340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:03.207617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:03.207896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:03.216839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:03.218272 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:03.227679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1275/2544] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:03.521937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:03.523599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:03.538210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1d7e9d-71e3-4c90-af98-e35df0c84a8c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:03.543559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:03.544458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:03.545041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:03.545470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] PATH_INFO: `/v3/projects/2309169db85e4ad19a5ba252fa776c8d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:03.546901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2309169db85e4ad19a5ba252fa776c8d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:03.549274 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:03.608273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2309169db85e4ad19a5ba252fa776c8d', 'request_id': 'req-2b1e1999-1ab2-4237-b9cc-4699fbb49105'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.618522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2309169db85e4ad19a5ba252fa776c8d', 'request_id': 'req-2b1e1999-1ab2-4237-b9cc-4699fbb49105'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.624631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2309169db85e4ad19a5ba252fa776c8d', 'request_id': 'req-2b1e1999-1ab2-4237-b9cc-4699fbb49105'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.639150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:03.643085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] The token cache is being invalidate because project 2309169db85e4ad19a5ba252fa776c8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:03.643565 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.644769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.650433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1270/2545] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:03 2022] DELETE /identity/v3/projects/2309169db85e4ad19a5ba252fa776c8d => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:03.656190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:03.668409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3697709a-5c58-4578-8c0b-9259b0e744c8 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:03.736763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:03.736763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:03.736763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:03.736763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] PATH_INFO: `/v3/users/3731dfa871da4831bfa34b43b83e833e` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:03.738157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3731dfa871da4831bfa34b43b83e833e)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:03.740164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:03.797195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3731dfa871da4831bfa34b43b83e833e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.807041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3731dfa871da4831bfa34b43b83e833e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.816906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3731dfa871da4831bfa34b43b83e833e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.839463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3731dfa871da4831bfa34b43b83e833e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.842002 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3b9a31d9-345a-44d9-a137-111d422a452e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3731dfa871da4831bfa34b43b83e833e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:03.850477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1276/2546] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:03 2022] DELETE /identity/v3/users/3731dfa871da4831bfa34b43b83e833e => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:04.154063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:04.156346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:04.170034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1e1999-1ab2-4237-b9cc-4699fbb49105 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:04.243373 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:04.244036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.244376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.244656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] PATH_INFO: `/v3/projects/e2f4f8a46fa746119cfde3da4213bde9` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.245903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2f4f8a46fa746119cfde3da4213bde9)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:04.247898 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:04.294438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2f4f8a46fa746119cfde3da4213bde9', 'request_id': 'req-8e28116a-abcd-45e5-82ac-c829d44ec3d6'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:04.302210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2f4f8a46fa746119cfde3da4213bde9', 'request_id': 'req-8e28116a-abcd-45e5-82ac-c829d44ec3d6'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:04.307697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2f4f8a46fa746119cfde3da4213bde9', 'request_id': 'req-8e28116a-abcd-45e5-82ac-c829d44ec3d6'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:04.318658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] The token cache is being invalidate because project e2f4f8a46fa746119cfde3da4213bde9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:04.318929 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:04.319597 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8e28116a-abcd-45e5-82ac-c829d44ec3d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:04.321470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1271/2547] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:04 2022] DELETE /identity/v3/projects/e2f4f8a46fa746119cfde3da4213bde9 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:04.326776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8e835-16c3-44e1-aca4-cd38a61e78e7 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.327289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8e835-16c3-44e1-aca4-cd38a61e78e7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.327289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d8e835-16c3-44e1-aca4-cd38a61e78e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.346724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-44d8e835-16c3-44e1-aca4-cd38a61e78e7 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:04.347984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-44d8e835-16c3-44e1-aca4-cd38a61e78e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:04.358224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:04.431227 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1277/2548] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:04.440230 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25794d-fabd-4cb3-887b-5d848874cb9c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.440791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25794d-fabd-4cb3-887b-5d848874cb9c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.441064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25794d-fabd-4cb3-887b-5d848874cb9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.471546 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-ed25794d-fabd-4cb3-887b-5d848874cb9c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:04.473270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ed25794d-fabd-4cb3-887b-5d848874cb9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:04.497264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8e835-16c3-44e1-aca4-cd38a61e78e7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:04.504244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1272/2549] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:04.509011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d8e835-16c3-44e1-aca4-cd38a61e78e7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:04.515670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:04.516681 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.517066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.518016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.518844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:04.519245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:04.522409 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:04.538108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1278/2550] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:04.543101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25794d-fabd-4cb3-887b-5d848874cb9c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:04.555117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25794d-fabd-4cb3-887b-5d848874cb9c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:04.560875 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:04.561547 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.561827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.562100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.562782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:04.564716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:04.605931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1273/2551] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:04 2022] POST /identity/v3/projects => generated 369 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:04.609752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:04.622020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84227415-cb0a-4439-a47d-12e82f6a3832 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:04.689100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:04.689714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.690000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.691207 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.691449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:04.693561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:04.728062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1279/2552] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:04 2022] POST /identity/v3/users => generated 285 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:04.728960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:04.742151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bad48f8-974e-4349-9f64-d0a3727af93c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:04.746858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:04.747541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.747824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.748098 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.748813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:04.750524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:04.762205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1274/2553] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:04 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:04.766443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:04.780628 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac8c8f9-895d-45cf-b5b2-8ecbaff29e0e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:04.786177 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3489fba7-8503-4e4d-933d-533ca5b82ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:04.786846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489fba7-8503-4e4d-933d-533ca5b82ded admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.787168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489fba7-8503-4e4d-933d-533ca5b82ded admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.787441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3489fba7-8503-4e4d-933d-533ca5b82ded admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.788203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3489fba7-8503-4e4d-933d-533ca5b82ded admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:04.789930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3489fba7-8503-4e4d-933d-533ca5b82ded admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:04.813079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1280/2554] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:04 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:04.817477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:04.829911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6ad3d7-400b-475f-a0b6-2418caa78b32 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:04.835456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:04.836241 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.836582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.837570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] PATH_INFO: `/v3/projects/4f0b85ab25654a92b29a3a589a646d59/users/a8fb74d95ce74a3f993efee6cc799e16/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.863100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f0b85ab25654a92b29a3a589a646d59, user_id=a8fb74d95ce74a3f993efee6cc799e16, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:04.867360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:04.895515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1275/2555] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:04 2022] PUT /identity/v3/projects/4f0b85ab25654a92b29a3a589a646d59/users/a8fb74d95ce74a3f993efee6cc799e16/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:04.908488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d386873-3cef-46d4-b21d-1ffd6433d89a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:04.908770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d386873-3cef-46d4-b21d-1ffd6433d89a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:04.909041 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d386873-3cef-46d4-b21d-1ffd6433d89a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:04.950788 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-8d386873-3cef-46d4-b21d-1ffd6433d89a None None] MFA Rules not processed for user `a8fb74d95ce74a3f993efee6cc799e16`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:04.952073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8d386873-3cef-46d4-b21d-1ffd6433d89a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:05.090407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1281/2556] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:04 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:05.100393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.163320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:05.216923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d1ce03-4ab2-4751-9a09-74078e0b8463 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.232593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.233792 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.237443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.237995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.240270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.242349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.307371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1276/2557] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:05 2022] POST /identity/v3/projects => generated 367 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:05.310171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d386873-3cef-46d4-b21d-1ffd6433d89a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.334175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d386873-3cef-46d4-b21d-1ffd6433d89a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.356700 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:05.440166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.441039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.441321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.441998 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.442892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.446169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.465769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.467908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:05.476086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1282/2558] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:05 2022] POST /identity/v3/users => generated 290 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:05.477410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.485093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12d71ea-d621-4807-8eee-a1c769c1d1f2 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.490483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.490973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cf8010-6cd3-4c6c-a91d-b1172b04a991 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.491549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.492175 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.492594 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] PATH_INFO: `/v3/users/46231c4eaa54470083424ac14dcff5a3` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.493990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] RBAC: Authorizing `identity:delete_user(user_id=46231c4eaa54470083424ac14dcff5a3)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.496039 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.496425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.496671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.496954 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.497224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.497940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.500346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.514528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1283/2559] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:05 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:05.520170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.538507 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0257c5e-b99f-4688-9f86-903281006967 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.544100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.545125 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.545658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.545943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] PATH_INFO: `/v3/projects/af9e6452167342108d41f812be717d4e/users/39ea9f35042d458bae54a9163ff22a40/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.561063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46231c4eaa54470083424ac14dcff5a3'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:05.563984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] RBAC: Authorizing `identity:create_grant(project_id=af9e6452167342108d41f812be717d4e, user_id=39ea9f35042d458bae54a9163ff22a40, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.566921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.583526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1284/2560] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:05 2022] PUT /identity/v3/projects/af9e6452167342108d41f812be717d4e/users/39ea9f35042d458bae54a9163ff22a40/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:05.586602 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46231c4eaa54470083424ac14dcff5a3'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:05.587768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.594402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46231c4eaa54470083424ac14dcff5a3'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:05.605200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46231c4eaa54470083424ac14dcff5a3'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:05.606344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-74289f39-4ced-425c-9826-1be07d65b99e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46231c4eaa54470083424ac14dcff5a3'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:05.614235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1277/2561] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:05 2022] DELETE /identity/v3/users/46231c4eaa54470083424ac14dcff5a3 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:05.619537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.620287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.621337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.643109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:05.644341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:05.671420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74158123-deaa-4bd6-9265-eba01729d11b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.702811 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1278/2562] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:05.712538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.714010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:05.727943 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.728575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.728962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.729391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.730101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.731756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.741330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:05.741562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1285/2563] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:05 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:05.745803 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.747504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:05.759575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9549636-866f-4360-a171-54d458868b12 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.764298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7478cdea-85ff-4ea3-b4ab-f850ef42e7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.764958 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7478cdea-85ff-4ea3-b4ab-f850ef42e7f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.765244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7478cdea-85ff-4ea3-b4ab-f850ef42e7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.765519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7478cdea-85ff-4ea3-b4ab-f850ef42e7f3 admin admin] PATH_INFO: `/v3/projects/af9e6452167342108d41f812be717d4e/users/39ea9f35042d458bae54a9163ff22a40/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.767490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7478cdea-85ff-4ea3-b4ab-f850ef42e7f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af9e6452167342108d41f812be717d4e, user_id=39ea9f35042d458bae54a9163ff22a40, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.770124 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7478cdea-85ff-4ea3-b4ab-f850ef42e7f3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.773174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b25f84-7c1b-4164-8f73-a9d20cfd12a1 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:05.773308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:05.777662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:05.778606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.778981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.779320 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.781745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1286/2564] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:05 2022] PUT /identity/v3/projects/af9e6452167342108d41f812be717d4e/users/39ea9f35042d458bae54a9163ff22a40/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:05.789210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:05.790703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:05.792970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:05.793266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:05.793536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:05.800757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1279/2565] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:05.823296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] MFA Rules not processed for user `39ea9f35042d458bae54a9163ff22a40`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:05.824705 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:05.899572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1287/2566] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:05 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:05.910587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:05.912457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:05.989578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c8c853-9ec2-4f93-b011-e73447d3cdaa service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:06.061396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:06.062127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:06.062466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:06.062803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:06.076376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:06.077870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:06.090444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1280/2567] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:05 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:06.117209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:06.119240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:06.199500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4e6093-4f01-4dde-97ef-629616abea68 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:06.204767 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:06.205588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:06.205872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:06.206146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] PATH_INFO: `/v3/projects/55313946954445649b97159fcde26a9e` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:06.207743 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55313946954445649b97159fcde26a9e)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:06.209682 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:06.267708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55313946954445649b97159fcde26a9e', 'request_id': 'req-5f2db6df-e4ed-4666-accd-c20056bb46e1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.275549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55313946954445649b97159fcde26a9e', 'request_id': 'req-5f2db6df-e4ed-4666-accd-c20056bb46e1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.281562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55313946954445649b97159fcde26a9e', 'request_id': 'req-5f2db6df-e4ed-4666-accd-c20056bb46e1'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.292832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] The token cache is being invalidate because project 55313946954445649b97159fcde26a9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:06.293116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.293789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.295362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1288/2568] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:06 2022] DELETE /identity/v3/projects/55313946954445649b97159fcde26a9e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:06.299387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:06.309977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e212fa-723b-4551-bbda-9edea3f0aa70 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:06.372791 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:06.373388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:06.373667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:06.373937 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] PATH_INFO: `/v3/users/c9d80a32a0cf4e0b99d7183bf06ce338` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:06.375392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9d80a32a0cf4e0b99d7183bf06ce338)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:06.377652 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:06.421149 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:06.441649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9d80a32a0cf4e0b99d7183bf06ce338'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.457243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9d80a32a0cf4e0b99d7183bf06ce338'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.464846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9d80a32a0cf4e0b99d7183bf06ce338'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.474409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9d80a32a0cf4e0b99d7183bf06ce338'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.475133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-4db2cd28-d101-4bef-a261-044f3dada467 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9d80a32a0cf4e0b99d7183bf06ce338'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:06.482911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1281/2569] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:06 2022] DELETE /identity/v3/users/c9d80a32a0cf4e0b99d7183bf06ce338 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:06.802359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:06.804359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:06.828359 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2db6df-e4ed-4666-accd-c20056bb46e1 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:06.847973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:06.915453 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:06.916205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:06.916483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:06.916763 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] PATH_INFO: `/v3/projects/5251e09424fd469f893cf9d29dcf7cfb` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:06.918055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5251e09424fd469f893cf9d29dcf7cfb)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:06.920688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.021341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5251e09424fd469f893cf9d29dcf7cfb', 'request_id': 'req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.040790 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5251e09424fd469f893cf9d29dcf7cfb', 'request_id': 'req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.047560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5251e09424fd469f893cf9d29dcf7cfb', 'request_id': 'req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.064440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] The token cache is being invalidate because project 5251e09424fd469f893cf9d29dcf7cfb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:07.064901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.066180 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f3b25c24-e4cc-422f-8aa2-070532bbd5a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.068585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1289/2570] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:06 2022] DELETE /identity/v3/projects/5251e09424fd469f893cf9d29dcf7cfb => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:07.073969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb98fb38-b419-4ec6-a04a-28a3cf80e81d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.074395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb98fb38-b419-4ec6-a04a-28a3cf80e81d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.074677 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb98fb38-b419-4ec6-a04a-28a3cf80e81d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.112233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-fb98fb38-b419-4ec6-a04a-28a3cf80e81d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:07.113763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fb98fb38-b419-4ec6-a04a-28a3cf80e81d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:07.235842 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1282/2571] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:07.237783 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8394784-b1dd-4ae5-8f78-954546b2b036 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.238068 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8394784-b1dd-4ae5-8f78-954546b2b036 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.238343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8394784-b1dd-4ae5-8f78-954546b2b036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.260807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-e8394784-b1dd-4ae5-8f78-954546b2b036 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:07.262271 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e8394784-b1dd-4ae5-8f78-954546b2b036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:07.274989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1290/2572] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:07.277404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb98fb38-b419-4ec6-a04a-28a3cf80e81d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.287333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb98fb38-b419-4ec6-a04a-28a3cf80e81d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.291965 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.292590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.292867 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.293142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.293849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:07.294151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.295855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.308352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1283/2573] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:07.312611 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8394784-b1dd-4ae5-8f78-954546b2b036 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.322619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8394784-b1dd-4ae5-8f78-954546b2b036 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.328671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.329307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.329599 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.329896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.330577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.332531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.361138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1291/2574] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/projects => generated 363 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:07.365077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.375357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8260c1-cb99-4883-b6c0-c7d79dbbc0d6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.440616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.441260 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.441682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.441968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.442683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.444771 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.480439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1284/2575] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/users => generated 282 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:07.483973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.498882 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c5b7f8-c490-4034-8625-6d3c6e829f1d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.505695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.506330 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.506680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.506964 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.508305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.510324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.524114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1292/2576] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:07 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:07.528024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.544024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc122238-ab97-4ed0-a14a-5eb48c96ac5e admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.550214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d4e00d-c216-4e38-9b1b-f78271469286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.550914 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d4e00d-c216-4e38-9b1b-f78271469286 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.551227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d4e00d-c216-4e38-9b1b-f78271469286 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.551502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d4e00d-c216-4e38-9b1b-f78271469286 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.553612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d4e00d-c216-4e38-9b1b-f78271469286 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.555612 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d4e00d-c216-4e38-9b1b-f78271469286 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.573467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.579848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1285/2577] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:07 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:07.590095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66939245-d4a9-4fd5-9acf-324561f9e09d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.597140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.597843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.598121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.598394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] PATH_INFO: `/v3/projects/6fe68ceee4af49f9a72c395d60c708c7/users/f501ba6fa81f4e188ab54841a3b03194/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.619799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6fe68ceee4af49f9a72c395d60c708c7, user_id=f501ba6fa81f4e188ab54841a3b03194, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.623747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.647533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1293/2578] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:07 2022] PUT /identity/v3/projects/6fe68ceee4af49f9a72c395d60c708c7/users/f501ba6fa81f4e188ab54841a3b03194/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:07.658225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231dd13e-79b3-42e9-8895-e73580f6f9a7 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.659054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231dd13e-79b3-42e9-8895-e73580f6f9a7 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.659580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231dd13e-79b3-42e9-8895-e73580f6f9a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.691141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:07.707183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-231dd13e-79b3-42e9-8895-e73580f6f9a7 None None] MFA Rules not processed for user `f501ba6fa81f4e188ab54841a3b03194`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:07.709555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-231dd13e-79b3-42e9-8895-e73580f6f9a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:07.777484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.779559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:07.834864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:07.835562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1286/2579] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:07.837520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231dd13e-79b3-42e9-8895-e73580f6f9a7 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.863941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5f4ef6-d8d4-4791-8d06-e7be0cfa9f62 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.870673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.872893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.873641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.873641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] PATH_INFO: `/v3/users/a8fb74d95ce74a3f993efee6cc799e16` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.874727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8fb74d95ce74a3f993efee6cc799e16)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.876750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.928061 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231dd13e-79b3-42e9-8895-e73580f6f9a7 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:07.936353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:07.937037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.937952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.938248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:07.939456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:07.940885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:07.941606 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8fb74d95ce74a3f993efee6cc799e16'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.950174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8fb74d95ce74a3f993efee6cc799e16'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.964212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8fb74d95ce74a3f993efee6cc799e16'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.979572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1287/2580] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/projects => generated 365 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:07.980185 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8fb74d95ce74a3f993efee6cc799e16'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.980723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:07.981201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6dc85e6f-14ea-4a8c-ac6f-691a36f0789d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8fb74d95ce74a3f993efee6cc799e16'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:07.982536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:07.991956 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1294/2581] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:07 2022] DELETE /identity/v3/users/a8fb74d95ce74a3f993efee6cc799e16 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:07.996905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:07.997433 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:07.998187 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.021979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:08.023241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.062452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49252160-97ff-41a6-9d02-de6cb988533c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.090852 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1295/2582] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:08.098682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:08.110365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.113791 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.131981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.132763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.133052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.133333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.134022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.141020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.173715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1288/2583] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:07 2022] POST /identity/v3/users => generated 289 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:08.178358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.190566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3c9dff-2d8c-4ea5-a5f7-3e6196b97a57 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.197157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.198021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.198437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.198717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.199467 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2feb72-1966-417b-b682-61d99e6ad125 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.199663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.202814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.207864 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.208736 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.209074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.209660 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.219112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1289/2584] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:08 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:08.220568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.222018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.223721 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.227877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:08.234052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1296/2585] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:08.236944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b577f7-6b2d-44f3-9bd7-e522d43f84b5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.243229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.244740 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.245245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.245741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] PATH_INFO: `/v3/projects/dbde09867fb941a79f7657d1ed1bb7b8/users/86f4d4b8f60844699fee3fa1e33f68ab/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.266483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbde09867fb941a79f7657d1ed1bb7b8, user_id=86f4d4b8f60844699fee3fa1e33f68ab, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.269497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.288255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1290/2586] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:08 2022] PUT /identity/v3/projects/dbde09867fb941a79f7657d1ed1bb7b8/users/86f4d4b8f60844699fee3fa1e33f68ab/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:08.293632 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.304200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16fc4d7-2c36-407c-abc6-213c692e4699 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.377086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.378103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.378596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.379152 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.380275 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.382410 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.395148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1297/2587] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:08 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:08.399571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.420170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad45f5e-38a7-4473-b1a3-19d7934f42cf admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.435583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cd5371-dac3-4880-9059-0f065027d691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.436033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd5371-dac3-4880-9059-0f065027d691 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.436355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd5371-dac3-4880-9059-0f065027d691 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.436633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cd5371-dac3-4880-9059-0f065027d691 admin admin] PATH_INFO: `/v3/projects/dbde09867fb941a79f7657d1ed1bb7b8/users/86f4d4b8f60844699fee3fa1e33f68ab/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.439497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd5371-dac3-4880-9059-0f065027d691 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbde09867fb941a79f7657d1ed1bb7b8, user_id=86f4d4b8f60844699fee3fa1e33f68ab, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.444737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cd5371-dac3-4880-9059-0f065027d691 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.475995 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1291/2588] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:08 2022] PUT /identity/v3/projects/dbde09867fb941a79f7657d1ed1bb7b8/users/86f4d4b8f60844699fee3fa1e33f68ab/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:08.478276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.480344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.487798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.488221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.488600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.524628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] MFA Rules not processed for user `86f4d4b8f60844699fee3fa1e33f68ab`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:08.526400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.559465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d999bc40-b8fc-4e98-b727-19b911681438 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.612288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.613212 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.613492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.613769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.616261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.618705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1292/2589] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:08 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:08.630204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.631627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.692744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48279c75-04c0-4f7d-94b9-a6451a5e9b53 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.696863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.697745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.698025 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.698297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.698448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.700205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.708316 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.709899 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.712820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1298/2590] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:08.714000 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.715708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.720978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1293/2591] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:08 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:08.725323 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:08.735692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45454679-4550-40e0-a8b0-c19dfe413f3f service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.748562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.750485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.751872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.752604 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] PATH_INFO: `/v3/projects/4f0b85ab25654a92b29a3a589a646d59` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.754181 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f0b85ab25654a92b29a3a589a646d59)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.757074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.807494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.812901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:08.820449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15f9d0-19c9-49ff-9051-7fea3dbb5e7e service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.837211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f0b85ab25654a92b29a3a589a646d59', 'request_id': 'req-42594444-e9db-451f-bd31-d862ecd3703d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:08.847402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f0b85ab25654a92b29a3a589a646d59', 'request_id': 'req-42594444-e9db-451f-bd31-d862ecd3703d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:08.854526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f0b85ab25654a92b29a3a589a646d59', 'request_id': 'req-42594444-e9db-451f-bd31-d862ecd3703d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:08.868458 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] The token cache is being invalidate because project 4f0b85ab25654a92b29a3a589a646d59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:08.868840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:08.869834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:08.874600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1299/2592] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:08 2022] DELETE /identity/v3/projects/4f0b85ab25654a92b29a3a589a646d59 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:08.875772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:08.877281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.890474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.892409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.893023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.893579 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.896179 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:08.952087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42594444-e9db-451f-bd31-d862ecd3703d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:08.957549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:08.958317 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:08.958656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:08.959019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] PATH_INFO: `/v3/users/39ea9f35042d458bae54a9163ff22a40` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:08.960286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] RBAC: Authorizing `identity:delete_user(user_id=39ea9f35042d458bae54a9163ff22a40)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.962581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:08.990765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:08.993466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:09.020803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1294/2593] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:08 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:09.067063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39ea9f35042d458bae54a9163ff22a40'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.085204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39ea9f35042d458bae54a9163ff22a40'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.100537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39ea9f35042d458bae54a9163ff22a40'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.116906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39ea9f35042d458bae54a9163ff22a40'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.117941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b46c2970-e5f9-4205-82a4-db86673e482b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39ea9f35042d458bae54a9163ff22a40'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.132781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1300/2594] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:08 2022] DELETE /identity/v3/users/39ea9f35042d458bae54a9163ff22a40 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:09.625571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:09.637562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21064e5b-2ae5-402f-a776-81d51fddbc11 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:09.701990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:09.702642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:09.702919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:09.703225 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] PATH_INFO: `/v3/projects/af9e6452167342108d41f812be717d4e` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:09.704396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] RBAC: Authorizing `identity:delete_project(project_id=af9e6452167342108d41f812be717d4e)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:09.706300 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:09.753509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af9e6452167342108d41f812be717d4e', 'request_id': 'req-7e52edb2-1457-496c-8410-086852863423'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.761396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af9e6452167342108d41f812be717d4e', 'request_id': 'req-7e52edb2-1457-496c-8410-086852863423'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.763554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:09.768881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af9e6452167342108d41f812be717d4e', 'request_id': 'req-7e52edb2-1457-496c-8410-086852863423'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.784302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] The token cache is being invalidate because project af9e6452167342108d41f812be717d4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:09.784578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.785305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7e52edb2-1457-496c-8410-086852863423 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:09.786967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1295/2595] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:09 2022] DELETE /identity/v3/projects/af9e6452167342108d41f812be717d4e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:09.792279 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01d12f1-e510-426f-9655-a85095175f44 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:09.792560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01d12f1-e510-426f-9655-a85095175f44 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:09.792830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01d12f1-e510-426f-9655-a85095175f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:09.801578 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:09.813885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c01d12f1-e510-426f-9655-a85095175f44 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:09.815448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c01d12f1-e510-426f-9655-a85095175f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:09.875773 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1301/2596] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:09.883619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc33aabe-6270-416a-9585-9dfb0cdb3ba8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:09.883908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc33aabe-6270-416a-9585-9dfb0cdb3ba8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:09.884185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc33aabe-6270-416a-9585-9dfb0cdb3ba8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:09.909940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-dc33aabe-6270-416a-9585-9dfb0cdb3ba8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:09.911235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-dc33aabe-6270-416a-9585-9dfb0cdb3ba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:09.919593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1296/2597] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:09.923916 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01d12f1-e510-426f-9655-a85095175f44 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:09.935195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01d12f1-e510-426f-9655-a85095175f44 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:09.941203 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:09.942328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:09.942622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:09.942876 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:09.943641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:09.943939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:09.945913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:09.963545 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1302/2598] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:18:09.964839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc33aabe-6270-416a-9585-9dfb0cdb3ba8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:09.975304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc33aabe-6270-416a-9585-9dfb0cdb3ba8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:09.980054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:09.980697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:09.980976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:09.981277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:09.982332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:09.984278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.019768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1297/2599] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:09 2022] POST /identity/v3/projects => generated 381 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:10.024224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.036044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d69cb-995d-46f7-8050-3c47f53877b7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.102067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.102703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.103007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.103319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.104019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:10.105967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.135693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1303/2600] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:10 2022] POST /identity/v3/users => generated 291 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:10.137099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.152210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96817dfd-d3d4-49c4-8863-ca1f6b870fe8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.158471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.158785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.159160 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.159493 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.160752 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:10.162326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.173024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1298/2601] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:10.177186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.187925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2808160b-519b-4317-a6ca-406c24474f8f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.193524 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273d2d1f-0291-4305-8ddd-92a456ed4fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.194157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d2d1f-0291-4305-8ddd-92a456ed4fec admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.194454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d2d1f-0291-4305-8ddd-92a456ed4fec admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.194722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273d2d1f-0291-4305-8ddd-92a456ed4fec admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.195825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273d2d1f-0291-4305-8ddd-92a456ed4fec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:10.197527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273d2d1f-0291-4305-8ddd-92a456ed4fec admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.211007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1304/2602] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:10.213379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.226921 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7fc75b-225a-4e89-8d87-91f61a02ce29 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.234283 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.234973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.235303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.235572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] PATH_INFO: `/v3/projects/5b5e18b1c2da4b56a8f078486f713e86/users/4046fd198eb44a2eb1242f66b362d03b/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.268777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b5e18b1c2da4b56a8f078486f713e86, user_id=4046fd198eb44a2eb1242f66b362d03b, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:10.271738 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.296167 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1299/2603] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:10 2022] PUT /identity/v3/projects/5b5e18b1c2da4b56a8f078486f713e86/users/4046fd198eb44a2eb1242f66b362d03b/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:10.300340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61132d-d7da-4f02-8e78-7729a288bd87 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.300741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61132d-d7da-4f02-8e78-7729a288bd87 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.301116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d61132d-d7da-4f02-8e78-7729a288bd87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.347394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-3d61132d-d7da-4f02-8e78-7729a288bd87 None None] MFA Rules not processed for user `4046fd198eb44a2eb1242f66b362d03b`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:10.348883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3d61132d-d7da-4f02-8e78-7729a288bd87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:10.440663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:10.446056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1305/2604] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:10 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:10.451896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.568557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27553cc8-b1cf-4fe8-aee9-993ee0ff82af admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.578053 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.579326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.580116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.580398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.581086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:10.584892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.639375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1300/2605] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:10 2022] POST /identity/v3/projects => generated 377 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:10.643598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d61132d-d7da-4f02-8e78-7729a288bd87 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.658482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d61132d-d7da-4f02-8e78-7729a288bd87 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.764442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.768342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.768635 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.768919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.769772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:10.772202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.829043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1306/2606] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:10 2022] POST /identity/v3/users => generated 295 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:10.833852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.852487 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c836d86-ef5b-448a-b7d2-2d5c5befd595 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.866177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.867216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.867500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.867773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:10.868492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:10.871831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:10.892041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1301/2607] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:10 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:10.896762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:10.920816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea22e49-cbb3-43df-92af-70a066b2b252 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:10.927102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:10.929310 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:10.931301 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:10.931854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] PATH_INFO: `/v3/projects/f5d487c5d1d14f48a6e108c377870501/users/f0a6ed28e39b46d0b51ddaa10d99747b/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:11.148874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5d487c5d1d14f48a6e108c377870501, user_id=f0a6ed28e39b46d0b51ddaa10d99747b, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:11.153745 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:11.174827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1307/2608] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:10 2022] PUT /identity/v3/projects/f5d487c5d1d14f48a6e108c377870501/users/f0a6ed28e39b46d0b51ddaa10d99747b/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:11.176156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:11.191994 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facd8037-97f2-445f-8b9d-cf08c041c299 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:11.459267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:11.467970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59559245-f7ba-4a5c-9d2e-753931a88327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:11.468779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59559245-f7ba-4a5c-9d2e-753931a88327 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:11.469356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59559245-f7ba-4a5c-9d2e-753931a88327 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:11.470029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59559245-f7ba-4a5c-9d2e-753931a88327 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:11.470790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59559245-f7ba-4a5c-9d2e-753931a88327 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:11.472684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59559245-f7ba-4a5c-9d2e-753931a88327 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:11.485956 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1302/2609] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:11 2022] GET /identity/v3/roles => generated 1420 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:11.487491 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:11.498162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07ea116-3855-4846-b70f-0c75ea4962bc admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:11.503096 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:11.503772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:11.504054 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:11.504331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] PATH_INFO: `/v3/projects/f5d487c5d1d14f48a6e108c377870501/users/f0a6ed28e39b46d0b51ddaa10d99747b/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:11.506280 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5d487c5d1d14f48a6e108c377870501, user_id=f0a6ed28e39b46d0b51ddaa10d99747b, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:11.508962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:11.519933 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1308/2610] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:11 2022] PUT /identity/v3/projects/f5d487c5d1d14f48a6e108c377870501/users/f0a6ed28e39b46d0b51ddaa10d99747b/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:11.531578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c6a040-7e4f-453c-a2a5-b8229a66ccc6 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:11.531800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c6a040-7e4f-453c-a2a5-b8229a66ccc6 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:11.532078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c6a040-7e4f-453c-a2a5-b8229a66ccc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:11.562195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-72c6a040-7e4f-453c-a2a5-b8229a66ccc6 None None] MFA Rules not processed for user `f0a6ed28e39b46d0b51ddaa10d99747b`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:11.563450 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-72c6a040-7e4f-453c-a2a5-b8229a66ccc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:11.667664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1303/2611] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:11 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:11.680562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:11.683022 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:11.768755 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74edda36-ba76-4cc1-ab3a-4fd9e1236aef admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:11.773380 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:11.774656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:11.778560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:11.778861 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:11.789170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:11.790525 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:11.800063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1309/2612] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:11 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:11.877614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c6a040-7e4f-453c-a2a5-b8229a66ccc6 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:11.887936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c6a040-7e4f-453c-a2a5-b8229a66ccc6 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:11.891396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:11.892016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:11.892289 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:11.892556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:11.969583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:11.971482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:11.983848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1304/2613] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:11 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:12.378264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:12.485068 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:12.485389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:12.491229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:12.495389 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbe8bd2-78c8-409a-bfe5-fba40c91acde service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:12.509328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffaac214-acb3-489d-98bf-fdb50a9cd7f8 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:12.515018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:12.516580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:12.517987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:12.518474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:12.519419 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] PATH_INFO: `/v3/users/4046fd198eb44a2eb1242f66b362d03b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:12.520048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:12.520691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:12.521188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] PATH_INFO: `/v3/users/f501ba6fa81f4e188ab54841a3b03194` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:12.521987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4046fd198eb44a2eb1242f66b362d03b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:12.524330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f501ba6fa81f4e188ab54841a3b03194)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:12.527421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:12.528052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:12.648370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4046fd198eb44a2eb1242f66b362d03b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.649482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f501ba6fa81f4e188ab54841a3b03194'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.658273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4046fd198eb44a2eb1242f66b362d03b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.669443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4046fd198eb44a2eb1242f66b362d03b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.671859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f501ba6fa81f4e188ab54841a3b03194'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.681071 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:12.684644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f501ba6fa81f4e188ab54841a3b03194'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.701618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f501ba6fa81f4e188ab54841a3b03194'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.701618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-99e36ef6-890f-4a75-a2f7-6fd9592180e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f501ba6fa81f4e188ab54841a3b03194'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.704804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4046fd198eb44a2eb1242f66b362d03b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.705591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a48c279-7ed6-4039-9731-d656f3f484b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4046fd198eb44a2eb1242f66b362d03b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:12.707802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1305/2614] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:12 2022] DELETE /identity/v3/users/f501ba6fa81f4e188ab54841a3b03194 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:12.712211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:12.712407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1310/2615] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:12 2022] DELETE /identity/v3/users/4046fd198eb44a2eb1242f66b362d03b => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:12.712608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:12.712990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:12.716897 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:12.717259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:12.717665 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:12.735261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:12.736782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:12.737486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:12.738669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:12.810977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1311/2616] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:12.812541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1306/2617] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:12.822305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:12.823997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:12.827271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:12.829560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:12.893776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c3a95a-b9cf-4085-8978-a1e8247e64af None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:12.897503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:12.898353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:12.898544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:12.898890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:12.903404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c530f27e-c5ab-43f6-b18c-b3bf29e78f92 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:12.907341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:12.908041 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:12.908393 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:12.908742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:12.909768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:12.911337 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:12.918912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:12.920462 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:12.922145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1312/2618] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:12.933091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1307/2619] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:13.555107 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:13.556708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:13.564464 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:13.566064 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:13.571786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7827bfb2-fb6e-433e-9d89-acdfe6826f38 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:13.577839 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:13.578711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:13.579080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:13.579421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] PATH_INFO: `/v3/projects/6fe68ceee4af49f9a72c395d60c708c7` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:13.580723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0544f0-8729-46a0-a606-9b086f25ef55 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:13.580835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6fe68ceee4af49f9a72c395d60c708c7)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:13.582754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:13.585873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:13.586521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:13.586801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:13.587111 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] PATH_INFO: `/v3/projects/5b5e18b1c2da4b56a8f078486f713e86` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:13.588527 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b5e18b1c2da4b56a8f078486f713e86)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:13.590685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:13.596201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:13.650815 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6fe68ceee4af49f9a72c395d60c708c7', 'request_id': 'req-8a98cb39-3709-40ff-915b-c7850c937673'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.666157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b5e18b1c2da4b56a8f078486f713e86', 'request_id': 'req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.672250 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6fe68ceee4af49f9a72c395d60c708c7', 'request_id': 'req-8a98cb39-3709-40ff-915b-c7850c937673'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.675969 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b5e18b1c2da4b56a8f078486f713e86', 'request_id': 'req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.679729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6fe68ceee4af49f9a72c395d60c708c7', 'request_id': 'req-8a98cb39-3709-40ff-915b-c7850c937673'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.684360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b5e18b1c2da4b56a8f078486f713e86', 'request_id': 'req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.693109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] The token cache is being invalidate because project 6fe68ceee4af49f9a72c395d60c708c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:13.693472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.694307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.696503 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] The token cache is being invalidate because project 5b5e18b1c2da4b56a8f078486f713e86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:13.696644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1313/2620] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:13 2022] DELETE /identity/v3/projects/6fe68ceee4af49f9a72c395d60c708c7 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:13.696729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.699686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.701913 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:13.703443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:13.704281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1308/2621] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:13 2022] DELETE /identity/v3/projects/5b5e18b1c2da4b56a8f078486f713e86 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:13.707880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:13.709412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:13.733334 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:13.793000 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd1527-e9cb-43bb-bf44-7ecfce4de15d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:13.797685 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a98cb39-3709-40ff-915b-c7850c937673 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:13.797873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:13.798719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:13.799051 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:13.799329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] PATH_INFO: `/v3/users/f0a6ed28e39b46d0b51ddaa10d99747b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:13.800507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0a6ed28e39b46d0b51ddaa10d99747b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:13.802560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:13.803171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:13.803884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:13.804273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:13.804824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] PATH_INFO: `/v3/users/86f4d4b8f60844699fee3fa1e33f68ab` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:13.811654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86f4d4b8f60844699fee3fa1e33f68ab)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:13.813858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:13.880056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0a6ed28e39b46d0b51ddaa10d99747b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.896200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86f4d4b8f60844699fee3fa1e33f68ab'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.899343 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0a6ed28e39b46d0b51ddaa10d99747b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.906450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86f4d4b8f60844699fee3fa1e33f68ab'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.907036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0a6ed28e39b46d0b51ddaa10d99747b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.916660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0a6ed28e39b46d0b51ddaa10d99747b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.917409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0a6ed28e39b46d0b51ddaa10d99747b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.924075 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86f4d4b8f60844699fee3fa1e33f68ab'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.925591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1309/2622] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:13 2022] DELETE /identity/v3/users/f0a6ed28e39b46d0b51ddaa10d99747b => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:13.962920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86f4d4b8f60844699fee3fa1e33f68ab'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.964080 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86f4d4b8f60844699fee3fa1e33f68ab'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:13.971411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1314/2623] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:13 2022] DELETE /identity/v3/users/86f4d4b8f60844699fee3fa1e33f68ab => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:14.436717 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.438835 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:14.453985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.455643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:14.492840 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:14.535820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03cfa85b-16b7-4adf-9ba1-199ce4539bd6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:14.552878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60da2da-b688-449d-b57d-0820e13a1ee9 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:14.558558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:14.559256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.559483 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.559784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] PATH_INFO: `/v3/projects/dbde09867fb941a79f7657d1ed1bb7b8` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.560960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbde09867fb941a79f7657d1ed1bb7b8)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:14.562896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:14.585883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:14.607960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:14.608705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.608984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.609255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] PATH_INFO: `/v3/projects/f5d487c5d1d14f48a6e108c377870501` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.610423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5d487c5d1d14f48a6e108c377870501)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:14.612383 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:14.613328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbde09867fb941a79f7657d1ed1bb7b8', 'request_id': 'req-30ca4050-525b-4200-bbb7-fb7be8c1710d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.623221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbde09867fb941a79f7657d1ed1bb7b8', 'request_id': 'req-30ca4050-525b-4200-bbb7-fb7be8c1710d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.629277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbde09867fb941a79f7657d1ed1bb7b8', 'request_id': 'req-30ca4050-525b-4200-bbb7-fb7be8c1710d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.642714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] The token cache is being invalidate because project dbde09867fb941a79f7657d1ed1bb7b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:14.643223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.644100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-30ca4050-525b-4200-bbb7-fb7be8c1710d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.651059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1315/2624] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:14 2022] DELETE /identity/v3/projects/dbde09867fb941a79f7657d1ed1bb7b8 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:14.657806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a96535-3fe7-47a4-8056-3db9a2fa382e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.658098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a96535-3fe7-47a4-8056-3db9a2fa382e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.658384 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a96535-3fe7-47a4-8056-3db9a2fa382e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.674977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5d487c5d1d14f48a6e108c377870501', 'request_id': 'req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.686018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-15a96535-3fe7-47a4-8056-3db9a2fa382e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:14.687291 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-15a96535-3fe7-47a4-8056-3db9a2fa382e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:14.687608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5d487c5d1d14f48a6e108c377870501', 'request_id': 'req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.695850 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5d487c5d1d14f48a6e108c377870501', 'request_id': 'req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.708355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] The token cache is being invalidate because project f5d487c5d1d14f48a6e108c377870501 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:14.708960 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.710056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aceb2cfb-7603-4de3-96fb-c6f9d50b4b70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:14.711773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1310/2625] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:14 2022] DELETE /identity/v3/projects/f5d487c5d1d14f48a6e108c377870501 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:14.719013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374aff71-825b-455d-9a91-7d4048fd93a8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.719768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374aff71-825b-455d-9a91-7d4048fd93a8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.719768 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374aff71-825b-455d-9a91-7d4048fd93a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.751611 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-374aff71-825b-455d-9a91-7d4048fd93a8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:14.752845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-374aff71-825b-455d-9a91-7d4048fd93a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:14.759785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1316/2626] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:14.771129 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1311/2627] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:14.771460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b135c635-465b-453f-9ce0-cbfb0dcf8568 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.771460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b135c635-465b-453f-9ce0-cbfb0dcf8568 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.771460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b135c635-465b-453f-9ce0-cbfb0dcf8568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.772798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c49d24-38db-4fdb-84cb-9a214df864f5 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.773210 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c49d24-38db-4fdb-84cb-9a214df864f5 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.773490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c49d24-38db-4fdb-84cb-9a214df864f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.790505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b135c635-465b-453f-9ce0-cbfb0dcf8568 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:14.791975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b135c635-465b-453f-9ce0-cbfb0dcf8568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:14.799161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-62c49d24-38db-4fdb-84cb-9a214df864f5 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:14.800342 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-62c49d24-38db-4fdb-84cb-9a214df864f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:14.803865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1317/2628] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:14.804813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b135c635-465b-453f-9ce0-cbfb0dcf8568 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.812731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1312/2629] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:14.814201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c49d24-38db-4fdb-84cb-9a214df864f5 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.817798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b135c635-465b-453f-9ce0-cbfb0dcf8568 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:14.826313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c49d24-38db-4fdb-84cb-9a214df864f5 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:14.829757 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:14.830518 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.830966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.831579 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.832394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:14.833122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:14.835376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:14.838750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:14.839751 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.841716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.842181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.843127 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:14.843839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:14.845833 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:14.849966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1318/2630] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:14.854598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.862562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1313/2631] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:14.866752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cf499e-dbd0-4fb9-be56-dce4744aefb2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:14.867325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.872267 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:14.873103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.873722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.874141 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.874986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:14.878508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:14.879623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1f7883-a9ab-44e6-9b82-ae76e683e48a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:14.884484 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:14.885239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:14.885522 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:14.885787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:14.886684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:14.888617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:14.916413 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1319/2632] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/projects => generated 385 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:14.920422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1314/2633] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/projects => generated 361 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:14.920770 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.923887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:14.931814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:15.018513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c2e950-34f6-4be8-ac26-da0d8e8ef837 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.020083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9582f1-46e0-461e-bed2-071b50a239a6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.023591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.024497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.024878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.025336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.025499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.027545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.027706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.027794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.027794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.027794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.028994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.041171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.055425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1320/2634] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/users => generated 293 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:15.059441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.073996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fca540c-8057-421c-97e1-60a01729b8de admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.083163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.084008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.085003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.085546 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.086462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.091369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1315/2635] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:14 2022] POST /identity/v3/users => generated 281 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:15.091534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.092586 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.104843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1321/2636] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:15 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:15.109466 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.110866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23330bed-ee31-40a1-84e3-7d68a280e995 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.115887 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.116524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.116859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.117141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.117847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.119568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.128601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b84c45-d6ae-4cd4-b96e-61779839ab31 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.134396 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1316/2637] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:15 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:15.135658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.136708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.137460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.137799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.138134 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.138939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.141686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:15.141974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.149076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb8b89-c6b1-41b8-93af-d80bfa4b5fe7 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.156472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.157560 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.157939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.158323 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.158735 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1322/2638] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:15 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:15.162837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.163883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.166336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.180527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f182ea2-f183-4a42-a804-0cdb2407a86e admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.187558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1317/2639] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:15 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:15.191031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.191228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78ad177-71f5-4413-b6b4-338caa19f993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.191949 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ad177-71f5-4413-b6b4-338caa19f993 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.192233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ad177-71f5-4413-b6b4-338caa19f993 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.192509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ad177-71f5-4413-b6b4-338caa19f993 admin admin] PATH_INFO: `/v3/projects/949f65d6f52e417a97b671a14cd86aca/users/82a5c7eb929646e78b6b6d35940d5d34/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.206078 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20a2213-7108-4a80-bed4-493bf639f649 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.212133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d2702d-f67d-4b45-8180-2ed7c587bae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.213099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d2702d-f67d-4b45-8180-2ed7c587bae7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.213382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d2702d-f67d-4b45-8180-2ed7c587bae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.213660 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d2702d-f67d-4b45-8180-2ed7c587bae7 admin admin] PATH_INFO: `/v3/projects/4fdd731415a9483381208855aacc87fb/users/3101d93a90ff43b9b313f21dffeb53da/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.219451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78ad177-71f5-4413-b6b4-338caa19f993 admin admin] RBAC: Authorizing `identity:create_grant(project_id=949f65d6f52e417a97b671a14cd86aca, user_id=82a5c7eb929646e78b6b6d35940d5d34, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.222798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78ad177-71f5-4413-b6b4-338caa19f993 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.236277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d2702d-f67d-4b45-8180-2ed7c587bae7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4fdd731415a9483381208855aacc87fb, user_id=3101d93a90ff43b9b313f21dffeb53da, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.240795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d2702d-f67d-4b45-8180-2ed7c587bae7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.251076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1323/2640] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:15 2022] PUT /identity/v3/projects/949f65d6f52e417a97b671a14cd86aca/users/82a5c7eb929646e78b6b6d35940d5d34/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:15.255558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc356e2-b851-4db5-825a-62d31d17893c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.256300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc356e2-b851-4db5-825a-62d31d17893c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.256642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc356e2-b851-4db5-825a-62d31d17893c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.271427 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1318/2641] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:15 2022] PUT /identity/v3/projects/4fdd731415a9483381208855aacc87fb/users/3101d93a90ff43b9b313f21dffeb53da/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:15.283952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.284375 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.284709 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.299526 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6dc356e2-b851-4db5-825a-62d31d17893c None None] MFA Rules not processed for user `82a5c7eb929646e78b6b6d35940d5d34`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:15.301415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6dc356e2-b851-4db5-825a-62d31d17893c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:15.336220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] MFA Rules not processed for user `3101d93a90ff43b9b313f21dffeb53da`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:15.337667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:15.436432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1324/2642] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:15.440775 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc356e2-b851-4db5-825a-62d31d17893c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.459873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1319/2643] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:15 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:15.497003 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.498906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:15.509812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:15.539474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc356e2-b851-4db5-825a-62d31d17893c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.545989 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.547044 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.547426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.547796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.548760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.551656 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.600538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d98aa-83fd-47f2-b3c9-cd1cf59f2058 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.604258 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1325/2644] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:15 2022] POST /identity/v3/projects => generated 387 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:15.605504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.606139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.606415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.606688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.608040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.621151 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.623035 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.634989 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1320/2645] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:15 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:15.650753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.676667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95940509-c6f6-4a7a-8167-b3c12fd9db74 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.728018 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd9fd60-dd19-4f54-9983-b57389c360bc admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.739932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.740581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.740866 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.741228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.744846 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.746879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.803434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1326/2646] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:15 2022] POST /identity/v3/users => generated 300 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:15.804716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.837781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf44b88-b11b-437b-bc80-061fcd4b43f7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.856355 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.857146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.857267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.857703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.857822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.858520 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.860150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.861094 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.861786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] PATH_INFO: `/v3/users/3101d93a90ff43b9b313f21dffeb53da` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.864091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.864552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3101d93a90ff43b9b313f21dffeb53da)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.874439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.878009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:15.894273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1327/2647] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:15 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:15.895380 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:15.923573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e29e1-7c93-4cec-a122-ab694f168b61 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:15.934644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:15.935610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:15.936278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:15.936557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] PATH_INFO: `/v3/projects/e93c59269a5f40b9a4ab55dd4dae6afa/users/27972566a3a14bb7a334dd461fff0b3b/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:15.969449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3101d93a90ff43b9b313f21dffeb53da'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:15.974023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e93c59269a5f40b9a4ab55dd4dae6afa, user_id=27972566a3a14bb7a334dd461fff0b3b, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:15.983084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:15.986786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3101d93a90ff43b9b313f21dffeb53da'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:16.003328 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3101d93a90ff43b9b313f21dffeb53da'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:16.025841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3101d93a90ff43b9b313f21dffeb53da'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:16.026195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1328/2648] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:15 2022] PUT /identity/v3/projects/e93c59269a5f40b9a4ab55dd4dae6afa/users/27972566a3a14bb7a334dd461fff0b3b/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:16.032173 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:16.034429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-d309fc39-731d-4ce1-a151-f9e229b184a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3101d93a90ff43b9b313f21dffeb53da'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:16.035318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:16.048569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1321/2649] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:15 2022] DELETE /identity/v3/users/3101d93a90ff43b9b313f21dffeb53da => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:16.050065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.050356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.050641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.073763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:16.075054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:16.145560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c14fdb9-acb2-4582-b946-a8abdb81a3f3 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:16.184151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:16.198684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1322/2650] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:16.212892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:16.218017 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:16.256759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:16.257559 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.258329 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.258601 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.259417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:16.262341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:16.281843 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1329/2651] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:16 2022] GET /identity/v3/roles => generated 1420 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:16.286072 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:16.297978 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b101a1e-8d12-41a1-94e6-a670a84eb371 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:16.298789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7a7327-032b-4056-a76c-a308a999ed85 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:16.302178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e740a-238b-4990-becf-bbefb28487de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:16.302941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e740a-238b-4990-becf-bbefb28487de service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.303360 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e740a-238b-4990-becf-bbefb28487de service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.303701 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e740a-238b-4990-becf-bbefb28487de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.307364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:16.308057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.308340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.308647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] PATH_INFO: `/v3/projects/e93c59269a5f40b9a4ab55dd4dae6afa/users/27972566a3a14bb7a334dd461fff0b3b/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.313716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e93c59269a5f40b9a4ab55dd4dae6afa, user_id=27972566a3a14bb7a334dd461fff0b3b, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:16.317010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e740a-238b-4990-becf-bbefb28487de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:16.318458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e740a-238b-4990-becf-bbefb28487de service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:16.328855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:16.329720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1323/2652] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:16.344430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1330/2653] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:16 2022] PUT /identity/v3/projects/e93c59269a5f40b9a4ab55dd4dae6afa/users/27972566a3a14bb7a334dd461fff0b3b/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:16.355700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.355700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.356095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.401614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] MFA Rules not processed for user `27972566a3a14bb7a334dd461fff0b3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:16.403296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:16.417006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:16.573810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1324/2654] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:16 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:16.591687 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:16.594668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:16.763027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0def8505-0168-4fd4-9eb8-7a855981ca40 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:16.771178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:16.772204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.772645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.773049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.786439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:16.788669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:16.803915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1331/2655] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:16 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:16.823154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:16.824783 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:16.923626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:16.935131 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e127268-ae28-48ba-b491-e03aa64096cf service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:16.935338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf251ea-d21e-4157-9a43-3244f076f3ef None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:16.939830 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:16.940593 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.940708 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:16.940931 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.941262 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] PATH_INFO: `/v3/users/82a5c7eb929646e78b6b6d35940d5d34` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.941459 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:16.941810 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:16.942155 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] PATH_INFO: `/v3/projects/4fdd731415a9483381208855aacc87fb` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:16.942684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82a5c7eb929646e78b6b6d35940d5d34)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:16.943782 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=4fdd731415a9483381208855aacc87fb)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:16.944670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:16.946330 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.005818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4fdd731415a9483381208855aacc87fb', 'request_id': 'req-1c86080a-1484-4805-a2e6-a5e46edd54ad'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.013290 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82a5c7eb929646e78b6b6d35940d5d34'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.019209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4fdd731415a9483381208855aacc87fb', 'request_id': 'req-1c86080a-1484-4805-a2e6-a5e46edd54ad'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.022436 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82a5c7eb929646e78b6b6d35940d5d34'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.026218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4fdd731415a9483381208855aacc87fb', 'request_id': 'req-1c86080a-1484-4805-a2e6-a5e46edd54ad'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.036274 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82a5c7eb929646e78b6b6d35940d5d34'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.039599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] The token cache is being invalidate because project 4fdd731415a9483381208855aacc87fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:17.039959 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.040839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-1c86080a-1484-4805-a2e6-a5e46edd54ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.042755 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1325/2656] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:16 2022] DELETE /identity/v3/projects/4fdd731415a9483381208855aacc87fb => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:17.047240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.047754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.048004 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.052417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82a5c7eb929646e78b6b6d35940d5d34'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.053238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-7ce8a553-0e97-49b2-850e-d434e4c4fd99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82a5c7eb929646e78b6b6d35940d5d34'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.064713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1332/2657] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:16 2022] DELETE /identity/v3/users/82a5c7eb929646e78b6b6d35940d5d34 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:17.066549 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.066938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.067309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.071808 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:17.073253 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.099597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:17.100831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.113655 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:17.129462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:17.154085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1326/2658] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:17.155029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.156227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.169468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46296e8-0c2a-43e7-8504-44f988b0e634 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.174975 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.175680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.175957 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.176228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] PATH_INFO: `/v3/users/8697f1ada4214e15876b2c2ec6184ed7` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.177520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=8697f1ada4214e15876b2c2ec6184ed7)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.179562 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.193032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1333/2659] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:17.208747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.210993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.241716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8697f1ada4214e15876b2c2ec6184ed7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.255892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8697f1ada4214e15876b2c2ec6184ed7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.265040 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8697f1ada4214e15876b2c2ec6184ed7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.274515 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8697f1ada4214e15876b2c2ec6184ed7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.275234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-11d6a3c1-ca7f-45bc-9e47-269013cd8aaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8697f1ada4214e15876b2c2ec6184ed7'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.276234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf39e7-812e-4ee0-884d-9ff3f9dc29b8 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.281204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.281844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.282226 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1327/2660] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:17 2022] DELETE /identity/v3/users/8697f1ada4214e15876b2c2ec6184ed7 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:17.282371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.282371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.284374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.286420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f585b5df-be0a-4786-b324-ff7dbc398887 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.286723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f585b5df-be0a-4786-b324-ff7dbc398887 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.287016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f585b5df-be0a-4786-b324-ff7dbc398887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.299462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.301162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.308357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f585b5df-be0a-4786-b324-ff7dbc398887 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:17.309904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f585b5df-be0a-4786-b324-ff7dbc398887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.313694 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1334/2661] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:17.398844 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1328/2662] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:17.403999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.417829 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83d899e-a853-4fee-ab19-fb19a766282d service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.425421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.431684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.432095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.432709 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.433750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:17.434462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.436475 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.452047 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1335/2663] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:17.456633 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f585b5df-be0a-4786-b324-ff7dbc398887 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.468666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f585b5df-be0a-4786-b324-ff7dbc398887 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.474058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.474702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.475031 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.475699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.476133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.478083 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.518892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1329/2664] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:17 2022] POST /identity/v3/projects => generated 377 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:17.524486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.540373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6befb44c-c22c-4a3d-9b23-e40961942dfe admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.553521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.555692 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.600824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:17.653294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba3991c-bf84-48e1-8769-c2d401eb2962 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.653432 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.654006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.654288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.654566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.655315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.657286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.661877 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.662582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.662859 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.663194 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] PATH_INFO: `/v3/projects/bd8a79690708464a9c3ffe381667a03c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.667502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd8a79690708464a9c3ffe381667a03c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.672087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.697156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1336/2665] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:17 2022] POST /identity/v3/users => generated 289 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:17.705781 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.718637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073654c-8f47-4a8b-82ea-2d1da8eb4cef admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.723944 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.724617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.724900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.725175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.725890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.727619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.735684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd8a79690708464a9c3ffe381667a03c', 'request_id': 'req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.740496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1337/2666] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:17 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:17.741407 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.744793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.746058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd8a79690708464a9c3ffe381667a03c', 'request_id': 'req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.752482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd8a79690708464a9c3ffe381667a03c', 'request_id': 'req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.757666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc33747-b818-4a3f-8f5d-3818367e6f44 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.765591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] The token cache is being invalidate because project bd8a79690708464a9c3ffe381667a03c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:17.765883 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.766545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8be4c3a3-17e5-4a21-bfe1-e7d98e90f825 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.768100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1330/2667] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:17 2022] DELETE /identity/v3/projects/bd8a79690708464a9c3ffe381667a03c => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:17.769404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.769687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.769959 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.788394 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] MFA Rules not processed for user `72614ab651224f4b9c0f2fb90806aadf`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:17.789592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.792098 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:17.817411 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.818062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.818340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.818617 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] PATH_INFO: `/v3/projects/949f65d6f52e417a97b671a14cd86aca` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.819818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] RBAC: Authorizing `identity:delete_project(project_id=949f65d6f52e417a97b671a14cd86aca)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.822306 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.848663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1331/2668] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:17 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:17.849824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.851510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.866555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c533a88-fe9d-467f-866e-2772592b3afd None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.886809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '949f65d6f52e417a97b671a14cd86aca', 'request_id': 'req-dba5349d-3702-4787-9d20-d1753556186d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.896395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '949f65d6f52e417a97b671a14cd86aca', 'request_id': 'req-dba5349d-3702-4787-9d20-d1753556186d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.903857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '949f65d6f52e417a97b671a14cd86aca', 'request_id': 'req-dba5349d-3702-4787-9d20-d1753556186d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.917588 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] The token cache is being invalidate because project 949f65d6f52e417a97b671a14cd86aca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:17.922580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.927886 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:17.929386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1338/2669] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:17 2022] DELETE /identity/v3/projects/949f65d6f52e417a97b671a14cd86aca => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:17.930477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.933626 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.948854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.950124 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.950408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.950680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:17.951456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:17.953106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:17.959023 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:17.965144 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1332/2670] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:17 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:17.970468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:17.973517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:17.991255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0dc32b-e9e8-4885-9e11-862022f12b35 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:17.997504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:17.998245 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:17.999109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:17.999109 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] PATH_INFO: `/v3/users/27972566a3a14bb7a334dd461fff0b3b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.000183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] RBAC: Authorizing `identity:delete_user(user_id=27972566a3a14bb7a334dd461fff0b3b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:18.002216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:18.027891 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba5349d-3702-4787-9d20-d1753556186d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:18.033926 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:18.034714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:18.035074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:18.035444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] PATH_INFO: `/v3/users/82fb3f03efb64adc84081706eb92eb9e` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.036782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82fb3f03efb64adc84081706eb92eb9e)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:18.039148 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:18.059558 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27972566a3a14bb7a334dd461fff0b3b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.073644 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27972566a3a14bb7a334dd461fff0b3b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.081649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27972566a3a14bb7a334dd461fff0b3b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.092756 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27972566a3a14bb7a334dd461fff0b3b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.093362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27972566a3a14bb7a334dd461fff0b3b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.097386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82fb3f03efb64adc84081706eb92eb9e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.100742 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1333/2671] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:17 2022] DELETE /identity/v3/users/27972566a3a14bb7a334dd461fff0b3b => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:18.101625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:18.103161 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:18.108043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82fb3f03efb64adc84081706eb92eb9e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.129342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82fb3f03efb64adc84081706eb92eb9e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.135412 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:18.151580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82fb3f03efb64adc84081706eb92eb9e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.154007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4db74d23-e887-4876-9cae-920c216ecfa6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82fb3f03efb64adc84081706eb92eb9e'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.163206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1339/2672] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:17 2022] DELETE /identity/v3/users/82fb3f03efb64adc84081706eb92eb9e => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:18.213449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f7856d-ace8-4319-b8a2-4425174b8933 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:18.296592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:18.297286 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:18.297568 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:18.298187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] PATH_INFO: `/v3/projects/03087e7d12d64f56bca9f0b5cd1775e0/users/f65f0239c286480b9c2025eba8c1178f/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.314795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03087e7d12d64f56bca9f0b5cd1775e0, user_id=f65f0239c286480b9c2025eba8c1178f, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:18.317653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:18.339070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1334/2673] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:18 2022] PUT /identity/v3/projects/03087e7d12d64f56bca9f0b5cd1775e0/users/f65f0239c286480b9c2025eba8c1178f/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:18.345151 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:18.345878 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:18.346357 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.383654 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] MFA Rules not processed for user `f65f0239c286480b9c2025eba8c1178f`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:18.385470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:18.483785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1340/2674] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:18 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:18.484839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:18.486917 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:18.500935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:18.539342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:18.540858 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:18.617387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955b165b-56f8-431f-a50f-8237fcfdb0d9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:18.622702 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:18.623564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:18.623685 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:18.624058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.624739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:18.626647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:18.703370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1335/2675] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:18 2022] POST /identity/v3/projects => generated 377 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:18.704613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:18.705368 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968ab3c9-9a3b-4d4e-9b80-0a52610d58ca None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:18.708591 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:18.789429 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:18.790344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:18.790692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:18.791360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] PATH_INFO: `/v3/projects/19282dfa2407442c95f4abc8b5a7e6a4` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.792457 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] RBAC: Authorizing `identity:delete_project(project_id=19282dfa2407442c95f4abc8b5a7e6a4)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:18.794473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:18.814760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583e6f5d-bc03-4908-a897-956358d0d713 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:18.824208 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:18.824852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:18.825135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:18.825403 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] PATH_INFO: `/v3/projects/e93c59269a5f40b9a4ab55dd4dae6afa` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.826831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e93c59269a5f40b9a4ab55dd4dae6afa)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:18.829018 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:18.839800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:18.874730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19282dfa2407442c95f4abc8b5a7e6a4', 'request_id': 'req-c2975de7-b5f4-48c4-94c9-0cb74408818d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.891487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19282dfa2407442c95f4abc8b5a7e6a4', 'request_id': 'req-c2975de7-b5f4-48c4-94c9-0cb74408818d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.904618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19282dfa2407442c95f4abc8b5a7e6a4', 'request_id': 'req-c2975de7-b5f4-48c4-94c9-0cb74408818d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.909235 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e93c59269a5f40b9a4ab55dd4dae6afa', 'request_id': 'req-14aa370b-0695-4328-8442-8a15c82f6d05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.921705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e93c59269a5f40b9a4ab55dd4dae6afa', 'request_id': 'req-14aa370b-0695-4328-8442-8a15c82f6d05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.928794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e93c59269a5f40b9a4ab55dd4dae6afa', 'request_id': 'req-14aa370b-0695-4328-8442-8a15c82f6d05'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.931492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] The token cache is being invalidate because project 19282dfa2407442c95f4abc8b5a7e6a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:18.931984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.933961 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.938168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1341/2676] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:18 2022] DELETE /identity/v3/projects/19282dfa2407442c95f4abc8b5a7e6a4 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:18.943794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] The token cache is being invalidate because project e93c59269a5f40b9a4ab55dd4dae6afa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:18.944251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.945666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-14aa370b-0695-4328-8442-8a15c82f6d05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:18.951082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1336/2677] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:18 2022] DELETE /identity/v3/projects/e93c59269a5f40b9a4ab55dd4dae6afa => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:18.952214 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:18.961625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c35ab3-605c-410e-b2df-dc848f9e5385 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:18.962120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c35ab3-605c-410e-b2df-dc848f9e5385 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:18.962581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c35ab3-605c-410e-b2df-dc848f9e5385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:18.964938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:18.988421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-17c35ab3-605c-410e-b2df-dc848f9e5385 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:18.989807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-17c35ab3-605c-410e-b2df-dc848f9e5385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:19.022616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:19.063884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2975de7-b5f4-48c4-94c9-0cb74408818d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.096674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1337/2678] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:19.098973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35f46a7-bc27-47ea-b57e-4447dc1c47bc None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.099259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35f46a7-bc27-47ea-b57e-4447dc1c47bc None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.099569 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35f46a7-bc27-47ea-b57e-4447dc1c47bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.124416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-f35f46a7-bc27-47ea-b57e-4447dc1c47bc None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:19.125876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f35f46a7-bc27-47ea-b57e-4447dc1c47bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:19.137481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.138232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.138586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.139216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.139996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.140126 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1338/2679] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:19.140994 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35f46a7-bc27-47ea-b57e-4447dc1c47bc None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.142713 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.151442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35f46a7-bc27-47ea-b57e-4447dc1c47bc None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.157453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.158324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.158578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.159077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.160264 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:19.160629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.162292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.176417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1342/2680] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:18 2022] POST /identity/v3/users => generated 289 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:19.179863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1339/2681] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:19.181002 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.181150 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.188736 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:19.191821 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e8bf03-f707-4029-8a2f-bb396eef9ab2 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.197502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.198150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.199270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.199827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.200748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.202848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.206160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89ba936-79ce-4ad1-bd06-631b51091db0 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.211974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.212643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.212970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.213244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.214081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.216007 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.229316 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1343/2682] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:19.233983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.245506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1340/2683] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/projects => generated 383 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:19.246458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.248398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5de141f-299e-469b-aea1-60f03026e2a8 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.287661 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:19.315638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.316273 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.316555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.316834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.316972 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33051d-f396-469e-bf0d-0a887d619d70 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.317540 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.319165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.323719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.325631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.326030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.326404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.327686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.329715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.333946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1344/2684] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:19.335113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.345600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3538b07f-f083-4efa-808c-c87426e361eb admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.350841 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.351715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.352081 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.352422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] PATH_INFO: `/v3/projects/c12e24933b8145d5815a9ea2c72c32b9/users/a1210279e7f048d2bbeaac62fc031159/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.355298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1341/2685] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/users => generated 292 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:19.360358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.371437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ca0998-ba29-49bf-8679-f0e3ee103535 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.371739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c12e24933b8145d5815a9ea2c72c32b9, user_id=a1210279e7f048d2bbeaac62fc031159, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.374459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.376580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33309fe4-95b3-4289-9d25-53355079e1fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.377946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33309fe4-95b3-4289-9d25-53355079e1fe admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.377946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33309fe4-95b3-4289-9d25-53355079e1fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.377946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33309fe4-95b3-4289-9d25-53355079e1fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.378911 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33309fe4-95b3-4289-9d25-53355079e1fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.380614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33309fe4-95b3-4289-9d25-53355079e1fe admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.391642 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1345/2686] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:19 2022] PUT /identity/v3/projects/c12e24933b8145d5815a9ea2c72c32b9/users/a1210279e7f048d2bbeaac62fc031159/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:19.399319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1342/2687] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:19.399438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.406044 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc6c8c1-37a8-43e5-8a22-0799b9c3485e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.406325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc6c8c1-37a8-43e5-8a22-0799b9c3485e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.406595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc6c8c1-37a8-43e5-8a22-0799b9c3485e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.437932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-5bc6c8c1-37a8-43e5-8a22-0799b9c3485e None None] MFA Rules not processed for user `a1210279e7f048d2bbeaac62fc031159`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:19.439185 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5bc6c8c1-37a8-43e5-8a22-0799b9c3485e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:19.474862 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742504a9-9494-48fc-9fca-a9f0292f0bd9 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.479622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.480249 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.480531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.480806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.481532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.483289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.485637 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:19.493443 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1346/2688] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:19.497777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.507193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fa1851-0741-41ba-9d02-1a6a99d3b289 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.507720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1343/2689] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:19.512369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc6c8c1-37a8-43e5-8a22-0799b9c3485e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.515045 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c3017-2a80-49f6-bb9e-4a0cf4ed7315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.515722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c3017-2a80-49f6-bb9e-4a0cf4ed7315 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.515995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c3017-2a80-49f6-bb9e-4a0cf4ed7315 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.516270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c3017-2a80-49f6-bb9e-4a0cf4ed7315 admin admin] PATH_INFO: `/v3/projects/b8a85dbb5c5644579e59ef11df25803c/users/68387f640bdb44219468bbba69317c0f/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.523218 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc6c8c1-37a8-43e5-8a22-0799b9c3485e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.528001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.528613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.528893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.529169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.529837 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.531764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.531880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c3017-2a80-49f6-bb9e-4a0cf4ed7315 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8a85dbb5c5644579e59ef11df25803c, user_id=68387f640bdb44219468bbba69317c0f, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.534405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3c3017-2a80-49f6-bb9e-4a0cf4ed7315 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.548804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1347/2690] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:19 2022] PUT /identity/v3/projects/b8a85dbb5c5644579e59ef11df25803c/users/68387f640bdb44219468bbba69317c0f/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:19.559977 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1344/2691] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:19.561470 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ba39f0-2574-4c59-9354-25ee0bddae78 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.561809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ba39f0-2574-4c59-9354-25ee0bddae78 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.562156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ba39f0-2574-4c59-9354-25ee0bddae78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.563564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.590464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-83ba39f0-2574-4c59-9354-25ee0bddae78 None None] MFA Rules not processed for user `68387f640bdb44219468bbba69317c0f`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:19.591703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-83ba39f0-2574-4c59-9354-25ee0bddae78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:19.626479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343e0c7-fcf6-4c00-a013-07e451cacb0b admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.631818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.632431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.632711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.632983 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.633645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.635631 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.654765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1348/2692] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:19.659667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ba39f0-2574-4c59-9354-25ee0bddae78 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.659928 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1345/2693] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/users => generated 295 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:19.663794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.673598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ba39f0-2574-4c59-9354-25ee0bddae78 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.675922 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd0218c-9d60-4797-a0af-7f591e29c30b admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.678711 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.679415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.679637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.679911 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.681012 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.681117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.681281 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.681802 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.682119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.682904 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.683269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.684839 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.694789 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:19.700305 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1346/2694] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:19.701133 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.722551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bf2d97-b9b6-48ad-966b-c071ff1ceeb7 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.727664 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.728348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.728735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.729221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] PATH_INFO: `/v3/projects/a7da1fd399794f3f9cc89c54a9dd0916/users/e5903dd224e3422caccaede3fb9bf09c/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.730309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1349/2695] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/projects => generated 385 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:19.736905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.748299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7da1fd399794f3f9cc89c54a9dd0916, user_id=e5903dd224e3422caccaede3fb9bf09c, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.751365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.769101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1347/2696] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:19 2022] PUT /identity/v3/projects/a7da1fd399794f3f9cc89c54a9dd0916/users/e5903dd224e3422caccaede3fb9bf09c/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:19.770180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.831014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74feaf43-f8cf-4937-a9fa-2b2d4cc58fd8 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.838197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.839087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.839551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.839939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b511fb7-184b-4abc-801a-f287b95116f7 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.840175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.840997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.843333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.845362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.845984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.846261 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.846535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.847262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.848852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.863010 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1348/2697] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:19.865553 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.877793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1350/2698] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/users => generated 293 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:19.878555 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68708dd-6887-4737-bda1-698567d2ca85 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.881577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:19.884204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bb8aa6-0bc3-4576-b7d5-2169bdc38422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.885027 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb8aa6-0bc3-4576-b7d5-2169bdc38422 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.885315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb8aa6-0bc3-4576-b7d5-2169bdc38422 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.885627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb8aa6-0bc3-4576-b7d5-2169bdc38422 admin admin] PATH_INFO: `/v3/projects/a7da1fd399794f3f9cc89c54a9dd0916/users/e5903dd224e3422caccaede3fb9bf09c/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.887944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bb8aa6-0bc3-4576-b7d5-2169bdc38422 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7da1fd399794f3f9cc89c54a9dd0916, user_id=e5903dd224e3422caccaede3fb9bf09c, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:19.890648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bb8aa6-0bc3-4576-b7d5-2169bdc38422 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:19.898593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:19.899498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402ef50b-cef8-4865-80c6-4d288f48426b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:19.905199 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1349/2699] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:19 2022] PUT /identity/v3/projects/a7da1fd399794f3f9cc89c54a9dd0916/users/e5903dd224e3422caccaede3fb9bf09c/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:19.914207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:19.914513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:19.914753 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:19.944812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] MFA Rules not processed for user `e5903dd224e3422caccaede3fb9bf09c`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:19.946442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:19.999116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:19.999643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.000061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.000340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.001055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.002748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.021737 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1351/2700] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:19 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:20.029553 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.043704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd46f38-f3dd-49a9-a516-afd444aa5eb2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.050760 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.051643 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.052027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.052398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.054175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.056607 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.061905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1350/2701] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:19 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:20.071386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1352/2702] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:20 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:20.073794 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.075382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:20.076228 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.088014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82528b99-0c40-4bed-9881-66c6ab737ea6 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.096318 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400cf022-c31d-48ef-bbb8-84b1d052d32c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.097106 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400cf022-c31d-48ef-bbb8-84b1d052d32c admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.097782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400cf022-c31d-48ef-bbb8-84b1d052d32c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.098086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400cf022-c31d-48ef-bbb8-84b1d052d32c admin admin] PATH_INFO: `/v3/projects/db4d0e7b5376487f95093793deb06b93/users/1935d9f13ad34c45b3ea68c937065ceb/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.121485 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400cf022-c31d-48ef-bbb8-84b1d052d32c admin admin] RBAC: Authorizing `identity:create_grant(project_id=db4d0e7b5376487f95093793deb06b93, user_id=1935d9f13ad34c45b3ea68c937065ceb, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.124487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400cf022-c31d-48ef-bbb8-84b1d052d32c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.139937 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1353/2703] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:20 2022] PUT /identity/v3/projects/db4d0e7b5376487f95093793deb06b93/users/1935d9f13ad34c45b3ea68c937065ceb/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:20.151074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6959c-eacb-489e-96ec-342e011f0a47 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.151669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6959c-eacb-489e-96ec-342e011f0a47 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.152169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6959c-eacb-489e-96ec-342e011f0a47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.162116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf274c3a-73b3-45f6-a9ea-b05e5e076873 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.183372 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-12b6959c-eacb-489e-96ec-342e011f0a47 None None] MFA Rules not processed for user `1935d9f13ad34c45b3ea68c937065ceb`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:20.184996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-12b6959c-eacb-489e-96ec-342e011f0a47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:20.240528 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.241392 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.241676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.242370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.244314 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:20.246340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:20.295042 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1354/2704] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:20.296623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6959c-eacb-489e-96ec-342e011f0a47 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.307600 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:20.310298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.312502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.323737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1351/2705] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:20 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:20.364800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6959c-eacb-489e-96ec-342e011f0a47 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.370170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.370917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.371292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.371636 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.372376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.374324 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.408741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1355/2706] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:20 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:20.409882 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.423893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae49aba-4906-4c3c-9b34-3d83db3f916b service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.499348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.500267 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.500623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.500879 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.501826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.504617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.534277 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1352/2707] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:20 2022] POST /identity/v3/users => generated 299 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:20.535633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.550137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ffc9d5-9c73-4f4d-9da3-b12fce3af862 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.557038 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.558200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.558704 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.559215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.560261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.562420 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.577035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1356/2708] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:20 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:20.578814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.593136 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d0814-6b93-4a7b-8ab5-1e0f381534f6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.598048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.598796 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.599092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.611513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] PATH_INFO: `/v3/projects/caabbdc923f3443f8662c4a7b4db060c/users/e3f91a4df60144a1a440556e3bf13907/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.630871 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] RBAC: Authorizing `identity:create_grant(project_id=caabbdc923f3443f8662c4a7b4db060c, user_id=e3f91a4df60144a1a440556e3bf13907, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.633590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.651251 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1353/2709] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:20 2022] PUT /identity/v3/projects/caabbdc923f3443f8662c4a7b4db060c/users/e3f91a4df60144a1a440556e3bf13907/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:20.655300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.665480 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f4612c-208c-412c-bb22-39f4b6d79dc5 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.724542 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2278510a-02eb-49ef-ae5f-829b2a913141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.725196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2278510a-02eb-49ef-ae5f-829b2a913141 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.725473 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2278510a-02eb-49ef-ae5f-829b2a913141 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.725738 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2278510a-02eb-49ef-ae5f-829b2a913141 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.726452 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2278510a-02eb-49ef-ae5f-829b2a913141 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.728087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2278510a-02eb-49ef-ae5f-829b2a913141 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.738002 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1357/2710] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:20 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:20.742437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.753395 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef30c16-8eda-4301-9dac-cbe431fbf17d admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.758341 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.759048 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.759332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.759608 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] PATH_INFO: `/v3/projects/caabbdc923f3443f8662c4a7b4db060c/users/e3f91a4df60144a1a440556e3bf13907/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.761736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=caabbdc923f3443f8662c4a7b4db060c, user_id=e3f91a4df60144a1a440556e3bf13907, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.764432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.776117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1354/2711] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:20 2022] PUT /identity/v3/projects/caabbdc923f3443f8662c4a7b4db060c/users/e3f91a4df60144a1a440556e3bf13907/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:20.786645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.786930 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.787243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.816263 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] MFA Rules not processed for user `e3f91a4df60144a1a440556e3bf13907`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:20.817471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:20.848392 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:20.889939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1358/2712] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:20 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:20.901262 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:20.960638 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff1886f-ff24-4d1f-b8e6-957fdf568bc6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:20.964337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:20.964959 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:20.965240 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:20.965512 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:20.975580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:20.977013 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:20.986352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1355/2713] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:20 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:21.523149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:21.524804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:21.541373 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e135bec-adb6-4385-abfa-5f7f99f5edd7 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:21.545596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:21.546220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:21.546495 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:21.546764 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:21.607853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:21.609237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:21.618200 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1359/2714] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:24.014492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:24.024147 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcad823-6574-4ce6-abc9-f9a8f1b51eb3 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:24.042065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:24.042065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:24.042065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:24.042065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:24.089757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:24.105905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:24.106536 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:24.116552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1356/2715] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:24 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:24.700183 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:24.710620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088d9c66-94fb-4738-a93f-04e0b3cc0d79 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:24.715597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:24.716241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:24.716522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:24.716797 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:24.785945 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:24.787472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:24.798890 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1360/2716] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:24 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:26.751102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:26.760106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cc347d-9e59-4c21-8ee1-c2c26886f416 service cinder] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:26.764145 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06db3c9-c2a0-4a97-9a3c-dd7adb11ff48 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:26.764735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06db3c9-c2a0-4a97-9a3c-dd7adb11ff48 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:26.765012 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06db3c9-c2a0-4a97-9a3c-dd7adb11ff48 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:26.765284 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06db3c9-c2a0-4a97-9a3c-dd7adb11ff48 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:26.824805 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06db3c9-c2a0-4a97-9a3c-dd7adb11ff48 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:26.826371 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06db3c9-c2a0-4a97-9a3c-dd7adb11ff48 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:26.836093 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1357/2717] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:26 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:28.762620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:28.823982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:28.829115 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c151fc5a-52f3-42b1-87d0-ad5a509636be service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:28.834497 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:28.835224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:28.835817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:28.836083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] PATH_INFO: `/v3/users/f65f0239c286480b9c2025eba8c1178f` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:28.837453 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f65f0239c286480b9c2025eba8c1178f)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:28.839650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:28.888750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f65f0239c286480b9c2025eba8c1178f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:28.896078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f65f0239c286480b9c2025eba8c1178f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:28.904837 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f65f0239c286480b9c2025eba8c1178f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:28.917390 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f65f0239c286480b9c2025eba8c1178f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:28.918202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f65f0239c286480b9c2025eba8c1178f'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:28.925537 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1361/2718] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:28 2022] DELETE /identity/v3/users/f65f0239c286480b9c2025eba8c1178f => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:28.930588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:28.931141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:28.931141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:28.951812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:28.953288 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.022273 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1358/2719] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:29.032831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:29.034471 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.052467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:29.053885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.065765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f385720-410d-4faf-ab57-c59c40b6364b None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:29.072077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:29.072737 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:29.073016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:29.073348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] PATH_INFO: `/v3/users/68387f640bdb44219468bbba69317c0f` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:29.074545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=68387f640bdb44219468bbba69317c0f)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:29.076976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:29.094215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8dfb4f-f4bd-4dbe-81fb-56ed882d9c4d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:29.098887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655bc8a1-c1da-4908-ac43-bb184a3ba4c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:29.099652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655bc8a1-c1da-4908-ac43-bb184a3ba4c3 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:29.100005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655bc8a1-c1da-4908-ac43-bb184a3ba4c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:29.100348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655bc8a1-c1da-4908-ac43-bb184a3ba4c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:29.111924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655bc8a1-c1da-4908-ac43-bb184a3ba4c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:29.113651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655bc8a1-c1da-4908-ac43-bb184a3ba4c3 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:29.124892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1362/2720] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:29.136372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68387f640bdb44219468bbba69317c0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.149797 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68387f640bdb44219468bbba69317c0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.158266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68387f640bdb44219468bbba69317c0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.166852 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68387f640bdb44219468bbba69317c0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.167745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68387f640bdb44219468bbba69317c0f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.170798 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:29.179425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1359/2721] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:29 2022] DELETE /identity/v3/users/68387f640bdb44219468bbba69317c0f => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:29.184458 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:29.184809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:29.185138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:29.205102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:29.207221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.273580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1363/2722] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:29.284204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:29.285624 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.389741 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28c443b-62aa-4f99-b6a1-8d33f82d75e7 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:29.394304 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:29.396191 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:29.396807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:29.397213 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:29.414162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:29.415923 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:29.427604 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1360/2723] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:29.543276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:29.544795 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.564208 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256a013-8e1e-4000-bf63-42770a6cb67e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:29.569381 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:29.570398 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:29.570670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:29.570939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] PATH_INFO: `/v3/projects/03087e7d12d64f56bca9f0b5cd1775e0` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:29.572255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=03087e7d12d64f56bca9f0b5cd1775e0)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:29.574232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:29.614097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:29.656569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03087e7d12d64f56bca9f0b5cd1775e0', 'request_id': 'req-f81b4121-ef5e-401e-86b7-9ea676c81c9d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.667703 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03087e7d12d64f56bca9f0b5cd1775e0', 'request_id': 'req-f81b4121-ef5e-401e-86b7-9ea676c81c9d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.673894 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03087e7d12d64f56bca9f0b5cd1775e0', 'request_id': 'req-f81b4121-ef5e-401e-86b7-9ea676c81c9d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.688205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] The token cache is being invalidate because project 03087e7d12d64f56bca9f0b5cd1775e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:29.688551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.689195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.690822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1364/2724] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:29 2022] DELETE /identity/v3/projects/03087e7d12d64f56bca9f0b5cd1775e0 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:29.695418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:29.706119 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0641085-7a31-4c78-bcb9-d896a1e00a10 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:29.765290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:29.766168 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:29.766517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:29.766795 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] PATH_INFO: `/v3/users/a1210279e7f048d2bbeaac62fc031159` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:29.768090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1210279e7f048d2bbeaac62fc031159)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:29.770029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:29.793364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:29.795303 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.824482 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81b4121-ef5e-401e-86b7-9ea676c81c9d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:29.829505 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:29.830427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:29.830847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:29.831591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] PATH_INFO: `/v3/projects/b8a85dbb5c5644579e59ef11df25803c` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:29.844171 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8a85dbb5c5644579e59ef11df25803c)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:29.846196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:29.849166 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1210279e7f048d2bbeaac62fc031159'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.863481 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1210279e7f048d2bbeaac62fc031159'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.871197 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1210279e7f048d2bbeaac62fc031159'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.881617 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1210279e7f048d2bbeaac62fc031159'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.882662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1210279e7f048d2bbeaac62fc031159'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.890988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1361/2725] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:29 2022] DELETE /identity/v3/users/a1210279e7f048d2bbeaac62fc031159 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:29.901160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8a85dbb5c5644579e59ef11df25803c', 'request_id': 'req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.911486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8a85dbb5c5644579e59ef11df25803c', 'request_id': 'req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.918376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8a85dbb5c5644579e59ef11df25803c', 'request_id': 'req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.933353 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] The token cache is being invalidate because project b8a85dbb5c5644579e59ef11df25803c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:29.933722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.934869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:29.936761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1365/2726] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:29 2022] DELETE /identity/v3/projects/b8a85dbb5c5644579e59ef11df25803c => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:29.941036 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:29.943308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:29.977092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:30.037105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82ce870-0b38-422e-9659-8ca6b37b5f83 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:30.042070 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:30.042818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:30.043187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:30.043529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] PATH_INFO: `/v3/users/1935d9f13ad34c45b3ea68c937065ceb` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:30.044793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1935d9f13ad34c45b3ea68c937065ceb)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:30.046787 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:30.111221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1935d9f13ad34c45b3ea68c937065ceb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.125526 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1935d9f13ad34c45b3ea68c937065ceb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.134861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1935d9f13ad34c45b3ea68c937065ceb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.145352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1935d9f13ad34c45b3ea68c937065ceb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.146209 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1935d9f13ad34c45b3ea68c937065ceb'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.154310 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1362/2727] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:29 2022] DELETE /identity/v3/users/1935d9f13ad34c45b3ea68c937065ceb => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:30.213023 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:30.214569 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:30.284448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b95e6-b7ad-48ea-bc5a-ddadff11cd53 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:30.314877 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:30.344474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:30.345136 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:30.345425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:30.345696 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] PATH_INFO: `/v3/projects/c12e24933b8145d5815a9ea2c72c32b9` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:30.346980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c12e24933b8145d5815a9ea2c72c32b9)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:30.348798 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:30.395716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c12e24933b8145d5815a9ea2c72c32b9', 'request_id': 'req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.404748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c12e24933b8145d5815a9ea2c72c32b9', 'request_id': 'req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.410622 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c12e24933b8145d5815a9ea2c72c32b9', 'request_id': 'req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.422934 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] The token cache is being invalidate because project c12e24933b8145d5815a9ea2c72c32b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:30.423762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.424445 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.426319 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1366/2728] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:30 2022] DELETE /identity/v3/projects/c12e24933b8145d5815a9ea2c72c32b9 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:30.430559 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:30.432292 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:30.448509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f024b3c3-ad83-46f1-b64d-fcbca3830ba9 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:30.478777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:30.481150 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:30.521950 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:30.523198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:30.523656 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:30.523970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] PATH_INFO: `/v3/users/e5903dd224e3422caccaede3fb9bf09c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:30.525296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5903dd224e3422caccaede3fb9bf09c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:30.527441 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:30.577011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:30.580011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5903dd224e3422caccaede3fb9bf09c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.580144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2d9b06-3f21-4c42-b6ba-4f34f349377a admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:30.584924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:30.585946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:30.585946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:30.586201 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] PATH_INFO: `/v3/projects/db4d0e7b5376487f95093793deb06b93` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:30.587387 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db4d0e7b5376487f95093793deb06b93)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:30.589451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:30.593369 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5903dd224e3422caccaede3fb9bf09c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.600349 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5903dd224e3422caccaede3fb9bf09c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.609421 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5903dd224e3422caccaede3fb9bf09c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.610039 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5903dd224e3422caccaede3fb9bf09c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.616367 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1363/2729] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:30 2022] DELETE /identity/v3/users/e5903dd224e3422caccaede3fb9bf09c => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:30.636600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db4d0e7b5376487f95093793deb06b93', 'request_id': 'req-d1713497-7c03-4e61-a36a-8f53217ea746'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.645204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db4d0e7b5376487f95093793deb06b93', 'request_id': 'req-d1713497-7c03-4e61-a36a-8f53217ea746'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.651641 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db4d0e7b5376487f95093793deb06b93', 'request_id': 'req-d1713497-7c03-4e61-a36a-8f53217ea746'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.653285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:30.668215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] The token cache is being invalidate because project db4d0e7b5376487f95093793deb06b93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:30.668488 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.669188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.671155 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1367/2730] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:30 2022] DELETE /identity/v3/projects/db4d0e7b5376487f95093793deb06b93 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:30.675674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:30.677112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:30.744953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de74c34-71fd-478e-849e-3788106e8881 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:30.809239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:30.809793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:30.810065 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:30.810338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] PATH_INFO: `/v3/users/e3f91a4df60144a1a440556e3bf13907` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:30.811543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3f91a4df60144a1a440556e3bf13907)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:30.813467 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:30.862222 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3f91a4df60144a1a440556e3bf13907'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.875548 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3f91a4df60144a1a440556e3bf13907'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.880302 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:30.883537 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3f91a4df60144a1a440556e3bf13907'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.895057 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:30.895593 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3f91a4df60144a1a440556e3bf13907'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.896092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3f91a4df60144a1a440556e3bf13907'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:30.897799 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:30.911159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1713497-7c03-4e61-a36a-8f53217ea746 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:30.911990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1364/2731] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:30 2022] DELETE /identity/v3/users/e3f91a4df60144a1a440556e3bf13907 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:30.969572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:30.970245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:30.970551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:30.970809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] PATH_INFO: `/v3/projects/a7da1fd399794f3f9cc89c54a9dd0916` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:30.972111 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7da1fd399794f3f9cc89c54a9dd0916)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:30.974282 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.028370 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7da1fd399794f3f9cc89c54a9dd0916', 'request_id': 'req-07f6c011-15b3-469f-83cf-bf086cb1b9b7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.036090 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7da1fd399794f3f9cc89c54a9dd0916', 'request_id': 'req-07f6c011-15b3-469f-83cf-bf086cb1b9b7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.042382 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7da1fd399794f3f9cc89c54a9dd0916', 'request_id': 'req-07f6c011-15b3-469f-83cf-bf086cb1b9b7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.053677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] The token cache is being invalidate because project a7da1fd399794f3f9cc89c54a9dd0916 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:31.053955 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.054597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-07f6c011-15b3-469f-83cf-bf086cb1b9b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.056178 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1368/2732] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:30 2022] DELETE /identity/v3/projects/a7da1fd399794f3f9cc89c54a9dd0916 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:31.218189 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:31.219935 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:31.235695 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b439ab09-7139-483d-85ad-f84096e5a391 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:31.331319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:31.331901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.332180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.332457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] PATH_INFO: `/v3/projects/caabbdc923f3443f8662c4a7b4db060c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.333801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] RBAC: Authorizing `identity:delete_project(project_id=caabbdc923f3443f8662c4a7b4db060c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:31.335849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.387322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'caabbdc923f3443f8662c4a7b4db060c', 'request_id': 'req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.395352 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'caabbdc923f3443f8662c4a7b4db060c', 'request_id': 'req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.401279 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'caabbdc923f3443f8662c4a7b4db060c', 'request_id': 'req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.412474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] The token cache is being invalidate because project caabbdc923f3443f8662c4a7b4db060c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:31.412838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.413606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7f3db2cb-35cd-4afd-8f4d-75492bbb901b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:31.419502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1365/2733] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:31 2022] DELETE /identity/v3/projects/caabbdc923f3443f8662c4a7b4db060c => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:31.421898 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c079a4-fa86-47fc-9a80-e2b75775d64c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.422257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c079a4-fa86-47fc-9a80-e2b75775d64c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.422609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c079a4-fa86-47fc-9a80-e2b75775d64c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.443364 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-91c079a4-fa86-47fc-9a80-e2b75775d64c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:31.444613 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-91c079a4-fa86-47fc-9a80-e2b75775d64c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:31.504255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1369/2734] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:31.574684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf4b494-d971-4019-b06d-2ac8c0925292 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.575072 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf4b494-d971-4019-b06d-2ac8c0925292 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.575415 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf4b494-d971-4019-b06d-2ac8c0925292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.598970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1bf4b494-d971-4019-b06d-2ac8c0925292 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:31.600183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1bf4b494-d971-4019-b06d-2ac8c0925292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:31.609848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1366/2735] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:31.613904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c079a4-fa86-47fc-9a80-e2b75775d64c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:31.624120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c079a4-fa86-47fc-9a80-e2b75775d64c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:31.632346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:31.632992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.633269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.633548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.634264 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:31.634574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:31.636325 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.649500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1370/2736] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:31.654186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf4b494-d971-4019-b06d-2ac8c0925292 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:31.666073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf4b494-d971-4019-b06d-2ac8c0925292 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:31.671457 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:31.672619 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.673054 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.673595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.674506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:31.676800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.709966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1367/2737] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:31 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:31.711724 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:31.723020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c06be8-3ec1-4891-8202-1df2cc945424 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:31.731739 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:31.789236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:31.789869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.790174 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.790501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.791259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:31.793902 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.821919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1371/2738] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:31 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:31.825849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:31.838326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb42ba3-e71d-45ba-9b3a-f0961e8f17e8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:31.845686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:31.846769 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.847090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.847767 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.849826 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:31.850641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.863832 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1368/2739] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:31.868078 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:31.879929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dab30c-9820-46ed-9e43-56757a0ab6b9 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:31.886126 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e77f69-7739-4061-a5cb-4916222e5463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:31.886777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e77f69-7739-4061-a5cb-4916222e5463 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.887116 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e77f69-7739-4061-a5cb-4916222e5463 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.887396 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e77f69-7739-4061-a5cb-4916222e5463 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.888121 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e77f69-7739-4061-a5cb-4916222e5463 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:31.889761 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e77f69-7739-4061-a5cb-4916222e5463 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.900248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1372/2740] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:31.904623 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:31.915091 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a82268f-4da3-4643-a822-be2abc9ee3e1 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:31.919813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:31.920490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.920765 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.921016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] PATH_INFO: `/v3/projects/d7eb31c3b2094f59a93489263a7f3c79/users/afc1d40a33074740a8e24a1651ff1715/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:31.940355 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7eb31c3b2094f59a93489263a7f3c79, user_id=afc1d40a33074740a8e24a1651ff1715, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:31.943140 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:31.962564 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1369/2741] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:31 2022] PUT /identity/v3/projects/d7eb31c3b2094f59a93489263a7f3c79/users/afc1d40a33074740a8e24a1651ff1715/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:31.974983 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b823f668-c10a-49f6-bd25-04d0067896de None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:31.975358 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b823f668-c10a-49f6-bd25-04d0067896de None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:31.975832 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b823f668-c10a-49f6-bd25-04d0067896de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.007388 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-b823f668-c10a-49f6-bd25-04d0067896de None None] MFA Rules not processed for user `afc1d40a33074740a8e24a1651ff1715`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:32.009786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-b823f668-c10a-49f6-bd25-04d0067896de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:32.087260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1373/2742] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:32.092115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.105730 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:32.156073 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a086389-c240-4c24-b0ec-003fb4de0851 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.161563 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.162196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.162482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.162763 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.163497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.165419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.203432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1370/2743] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:32 2022] POST /identity/v3/projects => generated 383 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:32.207805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b823f668-c10a-49f6-bd25-04d0067896de None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.218971 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b823f668-c10a-49f6-bd25-04d0067896de None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.275962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.276646 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.276924 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.277195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.277850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.279823 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.303016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1374/2744] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:32 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:32.307906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.318643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3437b73a-6ce0-4525-9c79-70361387ae84 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.323974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.324599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.324876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.325287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.325993 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.328290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.340059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1371/2745] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:32.344333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.354552 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78ab9c8-2cc7-449e-94c2-b64435071d3d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.359149 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cb3069-b064-4882-acbd-7069eba80859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.359771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cb3069-b064-4882-acbd-7069eba80859 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.360064 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cb3069-b064-4882-acbd-7069eba80859 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.360339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cb3069-b064-4882-acbd-7069eba80859 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.361049 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cb3069-b064-4882-acbd-7069eba80859 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.362686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cb3069-b064-4882-acbd-7069eba80859 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.364938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:32.377137 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1375/2746] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:32.378442 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.389346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf1c7d-8ccb-489c-a6e4-32eb24067498 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.394849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.395868 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.396243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.396587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] PATH_INFO: `/v3/projects/f1c77771c3524ad6987a127f9678e7de/users/21edcf6d9bbb43fd9ea96e93d56db2aa/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.418388 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1c77771c3524ad6987a127f9678e7de, user_id=21edcf6d9bbb43fd9ea96e93d56db2aa, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.421345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.441699 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1372/2747] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:32 2022] PUT /identity/v3/projects/f1c77771c3524ad6987a127f9678e7de/users/21edcf6d9bbb43fd9ea96e93d56db2aa/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:32.455623 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a1ad0-cf36-4a50-8763-c8270d606106 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.456076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a1ad0-cf36-4a50-8763-c8270d606106 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.456531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67a1ad0-cf36-4a50-8763-c8270d606106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.502144 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-c67a1ad0-cf36-4a50-8763-c8270d606106 None None] MFA Rules not processed for user `21edcf6d9bbb43fd9ea96e93d56db2aa`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:32.504215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c67a1ad0-cf36-4a50-8763-c8270d606106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:32.573677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1376/2748] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:32.578416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.641345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8790444e-ff89-4908-bd50-e5f8e0c2568a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.650598 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.651657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.651954 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.652254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.652896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.654823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.683903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1373/2749] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:32 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:32.688122 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67a1ad0-cf36-4a50-8763-c8270d606106 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.699845 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67a1ad0-cf36-4a50-8763-c8270d606106 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.755100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.755727 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.756005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.756278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.756978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.758975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.781034 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1377/2750] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:32 2022] POST /identity/v3/users => generated 299 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:32.785455 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.796498 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb66aa7-b78a-48ab-b8b5-a51259d7c3e5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.802331 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.802936 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.803237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.803516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.804250 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.805858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.816888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1374/2751] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:32.821535 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.832270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e888094-08ab-40c1-80d5-e45973e9d937 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.837776 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.838464 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.838816 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.839146 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] PATH_INFO: `/v3/projects/b6c809e787294e3caf7f362fe7295538/users/917624d442ea467a9159a8c70ba3c07d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.854828 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6c809e787294e3caf7f362fe7295538, user_id=917624d442ea467a9159a8c70ba3c07d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.857658 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.871812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1378/2752] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:32 2022] PUT /identity/v3/projects/b6c809e787294e3caf7f362fe7295538/users/917624d442ea467a9159a8c70ba3c07d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:32.876099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.887389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabb3468-2454-4ee8-b770-7e844e69909b admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.923552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:32.950058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b84105-3342-48a3-996c-959b42b35869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.950747 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b84105-3342-48a3-996c-959b42b35869 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.951062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b84105-3342-48a3-996c-959b42b35869 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.951336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b84105-3342-48a3-996c-959b42b35869 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.952043 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b84105-3342-48a3-996c-959b42b35869 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.953642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b84105-3342-48a3-996c-959b42b35869 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:32.967930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1375/2753] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:32.969528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:32.980638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b4620-ece1-4ab8-b86c-25c56c795228 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:32.986892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:32.987806 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:32.988164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:32.988477 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] PATH_INFO: `/v3/projects/b6c809e787294e3caf7f362fe7295538/users/917624d442ea467a9159a8c70ba3c07d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:32.990720 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6c809e787294e3caf7f362fe7295538, user_id=917624d442ea467a9159a8c70ba3c07d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:32.993566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:33.013091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1379/2754] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:32 2022] PUT /identity/v3/projects/b6c809e787294e3caf7f362fe7295538/users/917624d442ea467a9159a8c70ba3c07d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:33.023828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:33.024206 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:33.024490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:33.051880 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] MFA Rules not processed for user `917624d442ea467a9159a8c70ba3c07d`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:33.053097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:33.123838 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1376/2755] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:33 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:33.136490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:33.138256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:33.184129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:33.201168 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca91f1bc-ac79-4434-86a7-7b6c6694766c admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:33.204967 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:33.205580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:33.205857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:33.206127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:33.262577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:33.264104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:33.273534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1380/2756] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:34.557497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:34.559234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:34.689951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff1b403-ef06-4255-96ff-e5d176299213 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:34.690684 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=100 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:34.695060 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:34.695739 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:34.695985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:34.696303 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] PATH_INFO: `/v3/users/c70c080fc8354ccf90391984baf6052f` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:34.697669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] RBAC: Authorizing `identity:delete_user(user_id=c70c080fc8354ccf90391984baf6052f)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:34.699920 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:34.748353 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c70c080fc8354ccf90391984baf6052f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:34.763117 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c70c080fc8354ccf90391984baf6052f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:34.767578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c70c080fc8354ccf90391984baf6052f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:34.776814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c70c080fc8354ccf90391984baf6052f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:34.777519 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c70c080fc8354ccf90391984baf6052f'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:34.783582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1377/2757] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:34 2022] DELETE /identity/v3/users/c70c080fc8354ccf90391984baf6052f => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:34.798361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:34.817679 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b22c65-944c-438b-babb-2d33258631a3 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:34.879818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66b86d3-9369-49cb-b75e-4cd57e60e31b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:34.880441 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b86d3-9369-49cb-b75e-4cd57e60e31b service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:34.880716 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b86d3-9369-49cb-b75e-4cd57e60e31b service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:34.880985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b86d3-9369-49cb-b75e-4cd57e60e31b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:34.882717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f66b86d3-9369-49cb-b75e-4cd57e60e31b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:34.944118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66b86d3-9369-49cb-b75e-4cd57e60e31b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:34.946085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66b86d3-9369-49cb-b75e-4cd57e60e31b service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:34.956434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1381/2758] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:34 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:34.956847 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:34.958446 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:35.042248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119ab50a-b213-45d3-974e-63685986cfcc admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.046981 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.047626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.047910 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.048268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] PATH_INFO: `/v3/projects/8bd77269b8504984b3e086be66ac9713` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.049411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bd77269b8504984b3e086be66ac9713)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.051417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.100657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bd77269b8504984b3e086be66ac9713', 'request_id': 'req-c10c7164-03dd-4be0-a45e-895692bbcbbd'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:35.111007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bd77269b8504984b3e086be66ac9713', 'request_id': 'req-c10c7164-03dd-4be0-a45e-895692bbcbbd'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:35.117370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bd77269b8504984b3e086be66ac9713', 'request_id': 'req-c10c7164-03dd-4be0-a45e-895692bbcbbd'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:35.129244 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] The token cache is being invalidate because project 8bd77269b8504984b3e086be66ac9713 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:35.129531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:35.130184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c10c7164-03dd-4be0-a45e-895692bbcbbd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:35.132076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1378/2759] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:34 2022] DELETE /identity/v3/projects/8bd77269b8504984b3e086be66ac9713 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:35.137401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec45be-3152-4119-9406-1be9cc02480b None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.137689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec45be-3152-4119-9406-1be9cc02480b None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.137965 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ec45be-3152-4119-9406-1be9cc02480b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.178156 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:35.198277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-d9ec45be-3152-4119-9406-1be9cc02480b None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:35.199684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d9ec45be-3152-4119-9406-1be9cc02480b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:35.277768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1382/2760] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:35.289606 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf69cd6-b02f-49bd-b1ff-164e429b3e82 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.289976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf69cd6-b02f-49bd-b1ff-164e429b3e82 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.290404 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf69cd6-b02f-49bd-b1ff-164e429b3e82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.311282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:35.316932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-aaf69cd6-b02f-49bd-b1ff-164e429b3e82 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:35.318878 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-aaf69cd6-b02f-49bd-b1ff-164e429b3e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:35.327667 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1379/2761] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:35.331942 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ec45be-3152-4119-9406-1be9cc02480b None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.341883 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ec45be-3152-4119-9406-1be9cc02480b None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.347766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.348499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.348887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.349193 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.349906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:35.350209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.351867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.366587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1383/2762] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:18:35.371193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf69cd6-b02f-49bd-b1ff-164e429b3e82 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.381400 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf69cd6-b02f-49bd-b1ff-164e429b3e82 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.388561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.389171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.389451 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.389720 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.390409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.392326 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.423861 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1380/2763] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:35 2022] POST /identity/v3/projects => generated 397 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:35.428161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.439300 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed34e894-37fd-4932-a47b-75605fe04103 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.500326 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.500982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.501241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.501510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.502220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.504158 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.530851 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1384/2764] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:35 2022] POST /identity/v3/users => generated 299 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:35.533809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.545648 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692634a5-94c3-4de7-9528-6828f30b0eaa admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.551440 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.552141 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.552513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.552766 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.553466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.555174 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.565953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1381/2765] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:35 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:35.570338 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.581058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faa2547-699a-401a-9453-86326c7395bc admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.586403 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.587087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.587619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.587880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.588686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.590490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.604824 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1385/2766] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:35.606046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.616336 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dbbfc2-2890-4546-8b79-e3bea6a60e07 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.621132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34778ee5-b99b-4876-93a6-f67e01cccbb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.622042 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34778ee5-b99b-4876-93a6-f67e01cccbb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.622444 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34778ee5-b99b-4876-93a6-f67e01cccbb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.622800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34778ee5-b99b-4876-93a6-f67e01cccbb5 admin admin] PATH_INFO: `/v3/projects/ff812be404e54fa19262ff96832cd7eb/users/8ecfa772f3d64bea8ce2eb0a3b7a0981/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.637687 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34778ee5-b99b-4876-93a6-f67e01cccbb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff812be404e54fa19262ff96832cd7eb, user_id=8ecfa772f3d64bea8ce2eb0a3b7a0981, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.638906 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.640358 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34778ee5-b99b-4876-93a6-f67e01cccbb5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.640742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:35.654439 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1382/2767] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:35 2022] PUT /identity/v3/projects/ff812be404e54fa19262ff96832cd7eb/users/8ecfa772f3d64bea8ce2eb0a3b7a0981/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:35.666433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.667116 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.667423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.696362 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] MFA Rules not processed for user `8ecfa772f3d64bea8ce2eb0a3b7a0981`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:35.697584 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:35.703621 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11be45a0-ff73-43c5-a768-0ec64d3324d4 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.719929 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:35.753889 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.754519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.754802 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.755108 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.756904 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:35.765902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1383/2768] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:35 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:35.777401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:35.778745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:35.790649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49de61-1039-4251-b6b4-a68b4b13b26e None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:35.794345 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a4853-aa52-442e-9789-f6dcd39a0852 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:35.794986 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a4853-aa52-442e-9789-f6dcd39a0852 service cinder] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:35.795291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a4853-aa52-442e-9789-f6dcd39a0852 service cinder] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:35.795551 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a4853-aa52-442e-9789-f6dcd39a0852 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:35.803985 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756a4853-aa52-442e-9789-f6dcd39a0852 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.805413 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756a4853-aa52-442e-9789-f6dcd39a0852 service cinder] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.814029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1384/2769] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:35 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:35.821645 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:35.823140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:35.832976 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1386/2770] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:35 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:38.062024 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:38.062535 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:38.062641 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:38.093170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:38.094387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:38.164130 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1385/2771] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:18:38 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:38.213179 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:38.214939 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:38.300884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd48046-579e-4ef8-91c1-64a95a2d70f0 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:38.313596 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4372f0-b8bf-4f00-939b-314e0c38b38e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:38.314175 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4372f0-b8bf-4f00-939b-314e0c38b38e service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:38.314460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4372f0-b8bf-4f00-939b-314e0c38b38e service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:38.314740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4372f0-b8bf-4f00-939b-314e0c38b38e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:38.324765 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4372f0-b8bf-4f00-939b-314e0c38b38e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:38.326417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4372f0-b8bf-4f00-939b-314e0c38b38e service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:38.343147 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1387/2772] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:18:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:18:42.049057 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:42.051488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:42.085332 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:42.180294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20f0630-c8bc-4d6d-b948-7aa1ee52eb33 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:42.189082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:42.190222 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:42.190659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:42.191187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] PATH_INFO: `/v3/users/afc1d40a33074740a8e24a1651ff1715` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:42.192873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afc1d40a33074740a8e24a1651ff1715)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:42.196516 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:42.275777 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afc1d40a33074740a8e24a1651ff1715'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:42.293269 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afc1d40a33074740a8e24a1651ff1715'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:42.305784 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afc1d40a33074740a8e24a1651ff1715'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:42.318103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afc1d40a33074740a8e24a1651ff1715'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:42.318858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afc1d40a33074740a8e24a1651ff1715'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:42.329840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1386/2773] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:42 2022] DELETE /identity/v3/users/afc1d40a33074740a8e24a1651ff1715 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:42.331751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:42.332248 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:42.332650 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:42.355917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:42.357434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:42.433512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=100 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:42.447831 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1388/2774] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:42.460703 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:42.467001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:42.561422 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef57f23-e731-49dd-b639-906afc80f896 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:42.565239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:42.565890 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:42.566177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:42.566452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:42.576987 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:42.578432 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:42.588029 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1387/2775] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:42.925135 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:42.926995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:42.977893 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae956ea1-aaac-4b58-8131-4d4e2407d12e None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:42.990321 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:42.991281 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:42.998880 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:43.000399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] PATH_INFO: `/v3/projects/d7eb31c3b2094f59a93489263a7f3c79` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:43.002597 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7eb31c3b2094f59a93489263a7f3c79)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:43.006087 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:43.108103 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7eb31c3b2094f59a93489263a7f3c79', 'request_id': 'req-3d0f1abf-9034-428b-91e5-8bc6af1c8475'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.118575 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7eb31c3b2094f59a93489263a7f3c79', 'request_id': 'req-3d0f1abf-9034-428b-91e5-8bc6af1c8475'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.143809 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7eb31c3b2094f59a93489263a7f3c79', 'request_id': 'req-3d0f1abf-9034-428b-91e5-8bc6af1c8475'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.156066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] The token cache is being invalidate because project d7eb31c3b2094f59a93489263a7f3c79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:43.156513 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.157331 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.159074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1389/2776] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:42 2022] DELETE /identity/v3/projects/d7eb31c3b2094f59a93489263a7f3c79 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:43.163437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:43.174201 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772bbb39-114d-47f0-a3c0-03019315be0b service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:43.212735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:43.254764 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:43.255570 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:43.255902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:43.256157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] PATH_INFO: `/v3/users/21edcf6d9bbb43fd9ea96e93d56db2aa` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:43.257627 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21edcf6d9bbb43fd9ea96e93d56db2aa)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:43.259946 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:43.318092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21edcf6d9bbb43fd9ea96e93d56db2aa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.332187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21edcf6d9bbb43fd9ea96e93d56db2aa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.340095 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21edcf6d9bbb43fd9ea96e93d56db2aa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.349831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21edcf6d9bbb43fd9ea96e93d56db2aa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.350509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21edcf6d9bbb43fd9ea96e93d56db2aa'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.358359 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1388/2777] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:43 2022] DELETE /identity/v3/users/21edcf6d9bbb43fd9ea96e93d56db2aa => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:43.690938 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:43.692638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:43.729990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0f1abf-9034-428b-91e5-8bc6af1c8475 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:43.795826 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:43.798238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:43.798523 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:43.798793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] PATH_INFO: `/v3/projects/f1c77771c3524ad6987a127f9678e7de` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:43.801021 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1c77771c3524ad6987a127f9678e7de)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:43.802909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:43.862232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1c77771c3524ad6987a127f9678e7de', 'request_id': 'req-4a8f265f-7951-43ff-8f07-0b8f125ae72b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.873706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1c77771c3524ad6987a127f9678e7de', 'request_id': 'req-4a8f265f-7951-43ff-8f07-0b8f125ae72b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.880804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1c77771c3524ad6987a127f9678e7de', 'request_id': 'req-4a8f265f-7951-43ff-8f07-0b8f125ae72b'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.900879 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] The token cache is being invalidate because project f1c77771c3524ad6987a127f9678e7de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:43.901298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.903810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:43.906100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1390/2778] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:43 2022] DELETE /identity/v3/projects/f1c77771c3524ad6987a127f9678e7de => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:43.910419 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:43.911876 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:43.929399 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99f03fb-8dd0-481a-a0e1-4ee69cfc55a6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:44.017449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:44.019476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.019775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.020055 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] PATH_INFO: `/v3/users/917624d442ea467a9159a8c70ba3c07d` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.021270 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=917624d442ea467a9159a8c70ba3c07d)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:44.023774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:44.107540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '917624d442ea467a9159a8c70ba3c07d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.117948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:44.130001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '917624d442ea467a9159a8c70ba3c07d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.134544 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:44.136341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:44.139196 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '917624d442ea467a9159a8c70ba3c07d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.150620 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '917624d442ea467a9159a8c70ba3c07d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.155294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0adb1a5d-a6cc-433b-975b-d0a5d9103b65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '917624d442ea467a9159a8c70ba3c07d'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.165785 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8f265f-7951-43ff-8f07-0b8f125ae72b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:44.173293 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1389/2779] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:43 2022] DELETE /identity/v3/users/917624d442ea467a9159a8c70ba3c07d => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:44.185652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:44.255386 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:44.256211 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.256667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.257016 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] PATH_INFO: `/v3/users/8ecfa772f3d64bea8ce2eb0a3b7a0981` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.258374 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ecfa772f3d64bea8ce2eb0a3b7a0981)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:44.260600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:44.351492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ecfa772f3d64bea8ce2eb0a3b7a0981'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.360088 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ecfa772f3d64bea8ce2eb0a3b7a0981'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.370780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ecfa772f3d64bea8ce2eb0a3b7a0981'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.395215 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ecfa772f3d64bea8ce2eb0a3b7a0981'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.396349 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ecfa772f3d64bea8ce2eb0a3b7a0981'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.405532 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1391/2780] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:18:44 2022] DELETE /identity/v3/users/8ecfa772f3d64bea8ce2eb0a3b7a0981 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:44.413019 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.413378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.413681 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.439949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:44.441280 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:44.518814 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1390/2781] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:44.534629 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:44.537237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:44.625514 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:44.627134 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:44.641430 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcac06e-2106-46a8-b36c-70c675f0a7da admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:44.644234 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3041511d-51b1-46fe-ba24-6222a344ca40 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:44.646360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473775f4-6a94-4c04-add4-2065edd3955f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:44.647190 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473775f4-6a94-4c04-add4-2065edd3955f service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.648005 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473775f4-6a94-4c04-add4-2065edd3955f service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.648438 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473775f4-6a94-4c04-add4-2065edd3955f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.655056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:44.655781 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.656079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.656387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] PATH_INFO: `/v3/projects/b6c809e787294e3caf7f362fe7295538` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.657895 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6c809e787294e3caf7f362fe7295538)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:44.660001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:44.667261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473775f4-6a94-4c04-add4-2065edd3955f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:44.669293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473775f4-6a94-4c04-add4-2065edd3955f service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:44.681132 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1392/2782] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:44.741726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6c809e787294e3caf7f362fe7295538', 'request_id': 'req-e3291317-7693-4744-91e6-695e00515b9b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.755236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6c809e787294e3caf7f362fe7295538', 'request_id': 'req-e3291317-7693-4744-91e6-695e00515b9b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.765246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6c809e787294e3caf7f362fe7295538', 'request_id': 'req-e3291317-7693-4744-91e6-695e00515b9b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.781824 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] The token cache is being invalidate because project b6c809e787294e3caf7f362fe7295538 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:44.782103 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.783092 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-e3291317-7693-4744-91e6-695e00515b9b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:44.785387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1391/2783] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:44 2022] DELETE /identity/v3/projects/b6c809e787294e3caf7f362fe7295538 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:18:44.790484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2ea35-0587-49bb-9a6b-c9afd0cd23d0 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.791243 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2ea35-0587-49bb-9a6b-c9afd0cd23d0 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.791440 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2ea35-0587-49bb-9a6b-c9afd0cd23d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.818909 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-efa2ea35-0587-49bb-9a6b-c9afd0cd23d0 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:44.820516 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-efa2ea35-0587-49bb-9a6b-c9afd0cd23d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:44.857680 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:44.897925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1393/2784] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:18:44.905949 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7eea74-4122-49d8-81e9-b7e22c0e427f None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.906324 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7eea74-4122-49d8-81e9-b7e22c0e427f None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.906669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7eea74-4122-49d8-81e9-b7e22c0e427f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.948809 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-4d7eea74-4122-49d8-81e9-b7e22c0e427f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:44.950271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-4d7eea74-4122-49d8-81e9-b7e22c0e427f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:44.959056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1392/2785] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:44.963415 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa2ea35-0587-49bb-9a6b-c9afd0cd23d0 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:44.974708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa2ea35-0587-49bb-9a6b-c9afd0cd23d0 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:44.979734 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:44.980563 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:44.980997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:44.981533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:44.982449 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:18:44.982920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:44.984786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:44.998881 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1394/2786] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:18:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:18:45.002846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7eea74-4122-49d8-81e9-b7e22c0e427f None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.019275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7eea74-4122-49d8-81e9-b7e22c0e427f None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.024581 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.025431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.025869 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.026319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.027247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.029275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.071215 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1393/2787] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:18:45 2022] POST /identity/v3/projects => generated 369 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:18:45.072113 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.086401 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92045a69-0c22-449c-ba51-0c35d8bb7448 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.129572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.132397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:45.182288 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.182925 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.183255 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.183530 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.184418 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.186184 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:45.186638 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.215085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1395/2788] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:18:45 2022] POST /identity/v3/users => generated 285 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:18:45.219603 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.229984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23d1b3c-e8b0-4ec7-ac96-fd3400dd5907 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.236825 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.237691 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.237962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.238391 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.239356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb2c2a6-3973-4fb4-9e7a-eccfa7879980 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.239494 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.241238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.244510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.245195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.245478 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.245788 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] PATH_INFO: `/v3/projects/ff812be404e54fa19262ff96832cd7eb` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.247102 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff812be404e54fa19262ff96832cd7eb)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.249086 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.258277 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1396/2789] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:18:45.262787 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.277245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed22061f-a9a7-4105-bdab-f6533b2793d7 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.283444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.284435 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.284903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.285118 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.286019 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.287868 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.301548 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1397/2790] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:18:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:18:45.303443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff812be404e54fa19262ff96832cd7eb', 'request_id': 'req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:45.305669 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.314925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff812be404e54fa19262ff96832cd7eb', 'request_id': 'req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:45.319814 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4f39ed-fa32-43ef-ad4a-51578ffcaddd admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.323077 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff812be404e54fa19262ff96832cd7eb', 'request_id': 'req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:45.337940 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] The token cache is being invalidate because project ff812be404e54fa19262ff96832cd7eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:18:45.338229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:45.338903 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-0c931658-14d1-4e1c-afb2-0d50b5bc33c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:18:45.340574 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1394/2791] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:18:45 2022] DELETE /identity/v3/projects/ff812be404e54fa19262ff96832cd7eb => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:45.437222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b411a15-9304-463e-8381-779712144778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.438084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b411a15-9304-463e-8381-779712144778 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.438633 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b411a15-9304-463e-8381-779712144778 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.439062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b411a15-9304-463e-8381-779712144778 admin admin] PATH_INFO: `/v3/projects/229a86d61e1c472da0c4a32681df25af/users/7b0fb7785713405ab10401db480391b0/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.463668 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b411a15-9304-463e-8381-779712144778 admin admin] RBAC: Authorizing `identity:create_grant(project_id=229a86d61e1c472da0c4a32681df25af, user_id=7b0fb7785713405ab10401db480391b0, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.466940 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b411a15-9304-463e-8381-779712144778 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.497399 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1398/2792] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:18:45 2022] PUT /identity/v3/projects/229a86d61e1c472da0c4a32681df25af/users/7b0fb7785713405ab10401db480391b0/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:18:45.515760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.515760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.515760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.547583 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] MFA Rules not processed for user `7b0fb7785713405ab10401db480391b0`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:45.548888 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:45.666697 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1395/2793] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:18:45 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:45.670692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b411a15-9304-463e-8381-779712144778 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.681932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-4b411a15-9304-463e-8381-779712144778 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:45.691445 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:45.693022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:45.711725 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:45.755915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:45.772996 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783100e0-fcb6-4991-93b8-dd8b952c3cb8 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.778497 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d59d102-7d6d-4761-8cb4-e5b8908311ce service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.779416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d59d102-7d6d-4761-8cb4-e5b8908311ce service glance] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.779785 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d59d102-7d6d-4761-8cb4-e5b8908311ce service glance] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.780150 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d59d102-7d6d-4761-8cb4-e5b8908311ce service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.791006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d59d102-7d6d-4761-8cb4-e5b8908311ce service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.792626 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d59d102-7d6d-4761-8cb4-e5b8908311ce service glance] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.804523 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1396/2794] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:45 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:18:45.811476 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b411a15-9304-463e-8381-779712144778 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:45.816962 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:45.818063 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:45.818360 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:45.818637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:45.820834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:45.849686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:45.851344 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:45.864333 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1399/2795] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:18:45 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:18:55.799855 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:55.858260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=35 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:18:59.342939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:59.343431 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:59.343538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:59.363479 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:18:59.364700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:59.421841 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1397/2796] 10.222.0.42 () {58 vars in 967 bytes} [Tue Jul 26 03:18:59 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:18:59.434308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:18:59.436050 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:18:59.507196 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9aee3d-eabf-446a-a09b-c2f0e64f21cb service placement] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:18:59.515114 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2367723-11db-4871-a896-256b2288d64b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:18:59.515769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2367723-11db-4871-a896-256b2288d64b service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:18:59.516025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2367723-11db-4871-a896-256b2288d64b service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:18:59.516309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2367723-11db-4871-a896-256b2288d64b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:18:59.530872 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2367723-11db-4871-a896-256b2288d64b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:18:59.532459 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2367723-11db-4871-a896-256b2288d64b service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:18:59.540922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1400/2797] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:18:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:19:01.538534 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:01.540188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:01.606499 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a1004a-8204-470d-bc15-78b29fee5da3 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:01.611745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:01.612379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:01.612663 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:01.613076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] PATH_INFO: `/v3/users/72614ab651224f4b9c0f2fb90806aadf` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:01.614306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72614ab651224f4b9c0f2fb90806aadf)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:01.616298 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:01.666397 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72614ab651224f4b9c0f2fb90806aadf'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:01.678452 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72614ab651224f4b9c0f2fb90806aadf'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:01.685296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72614ab651224f4b9c0f2fb90806aadf'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:01.694429 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72614ab651224f4b9c0f2fb90806aadf'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:01.695120 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-7a80df3f-be0e-4982-88e6-a4f1fa320885 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72614ab651224f4b9c0f2fb90806aadf'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:01.701761 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1398/2798] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:19:01 2022] DELETE /identity/v3/users/72614ab651224f4b9c0f2fb90806aadf => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:01.881598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2367723-11db-4871-a896-256b2288d64b service swift] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:01.893859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2367723-11db-4871-a896-256b2288d64b service swift] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:01.951848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:01.952484 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:01.952759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:01.953027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] PATH_INFO: `/v3/projects/ad3b89efcbc94f19a867fe700046b0f2` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:01.954160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad3b89efcbc94f19a867fe700046b0f2)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:01.956097 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.001009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad3b89efcbc94f19a867fe700046b0f2', 'request_id': 'req-1b16419e-ecb6-4847-80ef-af5d9a27a352'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:02.009885 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad3b89efcbc94f19a867fe700046b0f2', 'request_id': 'req-1b16419e-ecb6-4847-80ef-af5d9a27a352'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:02.016352 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad3b89efcbc94f19a867fe700046b0f2', 'request_id': 'req-1b16419e-ecb6-4847-80ef-af5d9a27a352'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:02.027979 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] The token cache is being invalidate because project ad3b89efcbc94f19a867fe700046b0f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:19:02.028284 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:02.028996 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-1b16419e-ecb6-4847-80ef-af5d9a27a352 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:02.030667 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1401/2799] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:19:01 2022] DELETE /identity/v3/projects/ad3b89efcbc94f19a867fe700046b0f2 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:02.036118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b401bff8-391a-4437-8372-ca4f7f7fc450 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.036408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b401bff8-391a-4437-8372-ca4f7f7fc450 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.036680 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b401bff8-391a-4437-8372-ca4f7f7fc450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.057382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b401bff8-391a-4437-8372-ca4f7f7fc450 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:02.058540 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b401bff8-391a-4437-8372-ca4f7f7fc450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:02.078723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:02.121476 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1399/2800] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:02.129493 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e67786-316c-4827-929e-9166e5811466 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.129779 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e67786-316c-4827-929e-9166e5811466 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.130061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e67786-316c-4827-929e-9166e5811466 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.150434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-25e67786-316c-4827-929e-9166e5811466 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:02.151690 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-25e67786-316c-4827-929e-9166e5811466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:02.160717 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1402/2801] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:02.165045 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b401bff8-391a-4437-8372-ca4f7f7fc450 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:02.174812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b401bff8-391a-4437-8372-ca4f7f7fc450 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:02.180291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:02.180906 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.181186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.181454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.182153 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:19:02.182458 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:02.184106 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.197221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1400/2802] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:19:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:19:02.201777 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e67786-316c-4827-929e-9166e5811466 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:02.211315 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e67786-316c-4827-929e-9166e5811466 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:02.216242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:02.216946 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.217202 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.217501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.218188 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:02.220123 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.232499 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:02.252006 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1403/2803] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:19:02 2022] POST /identity/v3/projects => generated 367 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:19:02.256315 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:02.266689 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f73809d-032a-402d-a684-8e0c2aa99863 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:02.329381 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:02.329925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.330204 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.330473 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.331186 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:02.333112 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.354856 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1401/2804] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:19:02 2022] POST /identity/v3/users => generated 284 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:19:02.358993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:02.369595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2547020-7a2e-4b46-8899-d52e61eb1425 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:02.375058 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:02.375683 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.375963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.376236 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.376936 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:02.378555 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.388903 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1404/2805] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:19:02.392976 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:02.403988 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5bd0d6-1af2-4d45-b87a-9f46d467e024 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:02.409818 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ab9841-55da-4b89-9921-fcd9c3a9c794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:02.410435 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ab9841-55da-4b89-9921-fcd9c3a9c794 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.410719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ab9841-55da-4b89-9921-fcd9c3a9c794 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.410990 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ab9841-55da-4b89-9921-fcd9c3a9c794 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.411734 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ab9841-55da-4b89-9921-fcd9c3a9c794 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:02.413411 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ab9841-55da-4b89-9921-fcd9c3a9c794 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.424247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1402/2806] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:19:02.428465 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:02.439221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71774771-7c94-4ba8-b7d0-8b6d3be75fbc admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:02.445577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:02.446242 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.446567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.446820 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] PATH_INFO: `/v3/projects/0a32d58bd95b4045a564f3b3b163108c/users/0db45cd93afd498a87561635e1c41fc8/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.462028 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a32d58bd95b4045a564f3b3b163108c, user_id=0db45cd93afd498a87561635e1c41fc8, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:02.464782 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.479616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1405/2807] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:19:02 2022] PUT /identity/v3/projects/0a32d58bd95b4045a564f3b3b163108c/users/0db45cd93afd498a87561635e1c41fc8/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:02.491180 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.491477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.491754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.519311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] MFA Rules not processed for user `0db45cd93afd498a87561635e1c41fc8`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:02.520538 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:02.589827 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1403/2808] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:02 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:19:02.601527 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:02.603395 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:02.662095 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57214420-25ff-4833-a134-3336707787e2 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:02.666309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83462560-fd51-4467-8042-ebd1598266c3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:02.666993 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83462560-fd51-4467-8042-ebd1598266c3 service glance] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:02.667298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83462560-fd51-4467-8042-ebd1598266c3 service glance] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:02.667567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83462560-fd51-4467-8042-ebd1598266c3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:02.676522 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83462560-fd51-4467-8042-ebd1598266c3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:02.677953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83462560-fd51-4467-8042-ebd1598266c3 service glance] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:02.686912 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1406/2809] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:19:02 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:19:12.586915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=88,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:12.682754 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=44,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:17.027513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:17.029554 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:17.098700 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf8d475-c040-4d06-9f5f-934e1f38a23c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:17.103840 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:17.104483 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.104744 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.105022 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] PATH_INFO: `/v3/users/7b0fb7785713405ab10401db480391b0` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.106239 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b0fb7785713405ab10401db480391b0)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:17.108193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:17.159229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b0fb7785713405ab10401db480391b0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.171100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b0fb7785713405ab10401db480391b0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.178668 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b0fb7785713405ab10401db480391b0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.188973 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b0fb7785713405ab10401db480391b0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.189657 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b0fb7785713405ab10401db480391b0'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.196205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1404/2810] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:19:17 2022] DELETE /identity/v3/users/7b0fb7785713405ab10401db480391b0 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:17.201062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.201571 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.201695 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.220865 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:17.222069 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:17.284434 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1407/2811] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:17.294812 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:17.296566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:17.387402 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a3172d-c1db-49d2-ad69-d612c99407bb admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:17.392034 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8a502b-6d12-439b-b339-6e2126e1b920 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:17.392662 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8a502b-6d12-439b-b339-6e2126e1b920 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.392939 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8a502b-6d12-439b-b339-6e2126e1b920 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.393182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8a502b-6d12-439b-b339-6e2126e1b920 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.404135 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8a502b-6d12-439b-b339-6e2126e1b920 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:17.405607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8a502b-6d12-439b-b339-6e2126e1b920 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:17.415513 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1405/2812] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:19:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:19:17.686874 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:17.688547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:17.701003 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fafe42e-0872-417a-94ab-36d27890754f None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:17.705887 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:17.706539 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.706813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.707120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] PATH_INFO: `/v3/projects/229a86d61e1c472da0c4a32681df25af` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.708287 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] RBAC: Authorizing `identity:delete_project(project_id=229a86d61e1c472da0c4a32681df25af)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:17.710204 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:17.758655 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '229a86d61e1c472da0c4a32681df25af', 'request_id': 'req-beb12fcf-60f0-483a-999d-af660244b301'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.766920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '229a86d61e1c472da0c4a32681df25af', 'request_id': 'req-beb12fcf-60f0-483a-999d-af660244b301'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.772346 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '229a86d61e1c472da0c4a32681df25af', 'request_id': 'req-beb12fcf-60f0-483a-999d-af660244b301'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.783244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] The token cache is being invalidate because project 229a86d61e1c472da0c4a32681df25af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:19:17.783474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.784167 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-beb12fcf-60f0-483a-999d-af660244b301 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:17.785740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1408/2813] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:19:17 2022] DELETE /identity/v3/projects/229a86d61e1c472da0c4a32681df25af => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:17.791200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8efd50-73a8-4374-af5f-34f599499e8d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.791485 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8efd50-73a8-4374-af5f-34f599499e8d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.791757 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8efd50-73a8-4374-af5f-34f599499e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.800301 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:17.813958 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-8f8efd50-73a8-4374-af5f-34f599499e8d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:17.815549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8f8efd50-73a8-4374-af5f-34f599499e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:17.876115 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1406/2814] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:17.884400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed13e46-4802-48a3-940a-eb6497ea44f3 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.884686 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed13e46-4802-48a3-940a-eb6497ea44f3 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.884966 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed13e46-4802-48a3-940a-eb6497ea44f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.905093 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-1ed13e46-4802-48a3-940a-eb6497ea44f3 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:17.906309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-1ed13e46-4802-48a3-940a-eb6497ea44f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:17.915870 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1409/2815] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:17.920032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8efd50-73a8-4374-af5f-34f599499e8d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:17.930182 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8efd50-73a8-4374-af5f-34f599499e8d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:17.935203 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:17.935919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.936317 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.936658 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.937495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80427) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:19:17.937909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:17.939754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:17.954183 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1407/2816] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:19:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:19:17.958572 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed13e46-4802-48a3-940a-eb6497ea44f3 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:17.967985 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed13e46-4802-48a3-940a-eb6497ea44f3 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:17.972634 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:17.973241 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:17.973519 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:17.973984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:17.974670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:17.976616 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:18.008663 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1410/2817] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:19:17 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:19:18.013169 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:18.023333 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bab57c0-73fc-4db8-8ff6-e227aaa8a4e8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:18.081731 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:18.082389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:18.082610 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:18.082897 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:18.083671 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:18.085601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:18.109414 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1408/2818] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:19:18 2022] POST /identity/v3/users => generated 293 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:19:18.114234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:18.125259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d919fa4-7986-43d3-af3c-613d8bfa9e12 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:18.130307 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:18.130921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:18.131233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:18.131490 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:18.132189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:18.133813 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:18.144247 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1411/2819] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:19:18.148438 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:18.158885 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e66a589-f2c3-4ace-8e26-7314178480f8 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:18.164321 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022e6df7-513d-4593-bb82-b9b21a482fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:18.164947 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022e6df7-513d-4593-bb82-b9b21a482fb2 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:18.165221 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022e6df7-513d-4593-bb82-b9b21a482fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:18.165495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022e6df7-513d-4593-bb82-b9b21a482fb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:18.166195 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022e6df7-513d-4593-bb82-b9b21a482fb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:18.167862 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022e6df7-513d-4593-bb82-b9b21a482fb2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:18.177828 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1409/2820] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:19:18.182100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:18.192197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a59b21-ad02-45f1-a260-961bfab1ad58 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:18.197314 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:18.197970 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:18.198245 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:18.198521 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] PATH_INFO: `/v3/projects/ac6c4069e73d4966aa373c5cc84d8c00/users/219301d335c348358d8aaa63b86ddcc9/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:18.213043 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac6c4069e73d4966aa373c5cc84d8c00, user_id=219301d335c348358d8aaa63b86ddcc9, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:18.215733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:18.235919 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1412/2821] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:19:18 2022] PUT /identity/v3/projects/ac6c4069e73d4966aa373c5cc84d8c00/users/219301d335c348358d8aaa63b86ddcc9/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:18.243217 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b351a9d-9ec3-4b81-a254-5f5b0a5078ae None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:18.243607 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b351a9d-9ec3-4b81-a254-5f5b0a5078ae None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:18.243968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b351a9d-9ec3-4b81-a254-5f5b0a5078ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:18.269933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-1b351a9d-9ec3-4b81-a254-5f5b0a5078ae None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:18.271294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1b351a9d-9ec3-4b81-a254-5f5b0a5078ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:18.329580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1410/2822] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:18.339261 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:18.341294 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:18.365313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:18.402504 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd59a20-0f8c-41fc-9b41-9587cf50ec69 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:18.406238 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:18.406833 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:18.407154 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:18.407461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:18.417055 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:18.418528 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:18.428283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1413/2823] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:19:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:19:22.462170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71645c6-da26-43d9-a497-fa908efb63b4 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:22.462711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71645c6-da26-43d9-a497-fa908efb63b4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:22.462711 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71645c6-da26-43d9-a497-fa908efb63b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:22.483056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:22.493468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-b71645c6-da26-43d9-a497-fa908efb63b4 None None] MFA Rules not processed for user `219301d335c348358d8aaa63b86ddcc9`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:22.494786 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-b71645c6-da26-43d9-a497-fa908efb63b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:22.563384 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1411/2824] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:22.567577 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:22.578424 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624ff8d8-90a9-434f-bbfd-932138b6a775 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:22.586818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:22.586818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:22.586818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:22.586818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:22.587417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:22.587417 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:22.620610 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1414/2825] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:19:22 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:19:22.621647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71645c6-da26-43d9-a497-fa908efb63b4 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:22.632917 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71645c6-da26-43d9-a497-fa908efb63b4 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:22.689845 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:22.690517 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:22.690799 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:22.691118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:22.691793 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:22.694087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:22.719433 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1412/2826] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:19:22 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:19:22.723582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:22.735094 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668eae92-9540-478d-a95c-d3825a88a4aa admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:22.740260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:22.740884 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:22.741166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:22.741462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:22.742170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:22.743848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:22.754819 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1415/2827] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 26 03:19:22.759216 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:22.770096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e91d9e6-4521-4619-a320-fcd6d21f9ce6 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:22.775325 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:22.776008 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:22.776299 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:22.776571 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] PATH_INFO: `/v3/projects/c5fdb1d2ed3c4ab4977d81294e5d821f/users/61b086c34a114c06b35bd2b54a55014d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:22.792122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5fdb1d2ed3c4ab4977d81294e5d821f, user_id=61b086c34a114c06b35bd2b54a55014d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:22.794807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:22.810076 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1413/2828] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:19:22 2022] PUT /identity/v3/projects/c5fdb1d2ed3c4ab4977d81294e5d821f/users/61b086c34a114c06b35bd2b54a55014d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:22.814285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:22.825511 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bde7de6-1f97-4814-9ae0-f9dace1186ee admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:22.896771 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e53e29-6646-446e-a49e-75bad55a938b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:22.897305 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e53e29-6646-446e-a49e-75bad55a938b admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:22.897584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e53e29-6646-446e-a49e-75bad55a938b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:22.897854 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e53e29-6646-446e-a49e-75bad55a938b admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:22.898531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e53e29-6646-446e-a49e-75bad55a938b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:22.900172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e53e29-6646-446e-a49e-75bad55a938b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:22.910162 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1416/2829] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:22 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:19:22.914572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:22.925007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69702e1b-fa63-4d6a-b038-a19032fd7d63 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:22.930058 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:22.930716 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:22.930984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:22.931297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] PATH_INFO: `/v3/projects/c5fdb1d2ed3c4ab4977d81294e5d821f/users/61b086c34a114c06b35bd2b54a55014d/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:22.933357 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5fdb1d2ed3c4ab4977d81294e5d821f, user_id=61b086c34a114c06b35bd2b54a55014d, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:22.936146 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:22.947162 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1414/2830] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:19:22 2022] PUT /identity/v3/projects/c5fdb1d2ed3c4ab4977d81294e5d821f/users/61b086c34a114c06b35bd2b54a55014d/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:19:27.523139 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1feb9ab-5f53-4c6b-9e3e-3873f0e24497 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:27.523582 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1feb9ab-5f53-4c6b-9e3e-3873f0e24497 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:27.523857 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1feb9ab-5f53-4c6b-9e3e-3873f0e24497 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:27.566994 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-e1feb9ab-5f53-4c6b-9e3e-3873f0e24497 None None] MFA Rules not processed for user `61b086c34a114c06b35bd2b54a55014d`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:27.568442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e1feb9ab-5f53-4c6b-9e3e-3873f0e24497 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:27.710697 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1417/2831] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:27 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:27.725714 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:27.727556 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:27.807090 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cae170f-3536-4e71-a058-16e7d9e7435c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:27.811443 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81ec7fc0b48643739402efba1285c713', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:27.812409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:27.812848 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:27.813268 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:27.818456 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:27.887492 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:27.889154 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:27.899142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1415/2832] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:19:27 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:19:28.064232 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:28.064557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:28.064835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:28.085980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:28.087259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:28.093707 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:28.153953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1418/2833] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:19:28 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:28.166338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:28.167881 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:28.238490 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e5ca5c-3dac-4b99-9a1e-a71b5df160eb service glance] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:28.243573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5171c8-9b6e-471a-9df2-f1c044421d5e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:28.244193 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5171c8-9b6e-471a-9df2-f1c044421d5e service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:28.244472 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5171c8-9b6e-471a-9df2-f1c044421d5e service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:28.244713 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5171c8-9b6e-471a-9df2-f1c044421d5e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:28.252634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5171c8-9b6e-471a-9df2-f1c044421d5e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:28.254229 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5171c8-9b6e-471a-9df2-f1c044421d5e service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:28.261948 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1416/2834] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:19:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:19:35.818195 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:35.820091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:35.905052 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79384d4f-c8c0-4938-b366-5d3a15a86f95 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:35.909778 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:35.910460 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:35.910990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:35.911257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] PATH_INFO: `/v3/users/0db45cd93afd498a87561635e1c41fc8` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:35.912576 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0db45cd93afd498a87561635e1c41fc8)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:35.914584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:35.974999 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0db45cd93afd498a87561635e1c41fc8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:35.983014 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0db45cd93afd498a87561635e1c41fc8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:35.993166 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0db45cd93afd498a87561635e1c41fc8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.015461 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0db45cd93afd498a87561635e1c41fc8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.016298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0db45cd93afd498a87561635e1c41fc8'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.023376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1419/2835] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:19:35 2022] DELETE /identity/v3/users/0db45cd93afd498a87561635e1c41fc8 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:36.027905 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.028418 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.028520 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.052306 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:36.053530 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:36.119750 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1417/2836] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:19:36.130037 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:36.131671 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:36.180404 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:36.200836 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1baf41-4ce2-44f6-8d7f-8b56420d4b1f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:36.204922 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822a9223-306c-4fc0-ba8f-aa0bd8646059 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:36.205609 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a9223-306c-4fc0-ba8f-aa0bd8646059 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.205867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a9223-306c-4fc0-ba8f-aa0bd8646059 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.206138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822a9223-306c-4fc0-ba8f-aa0bd8646059 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.216684 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822a9223-306c-4fc0-ba8f-aa0bd8646059 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:36.218299 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822a9223-306c-4fc0-ba8f-aa0bd8646059 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:36.235198 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1420/2837] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:19:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:19:36.522009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:36.523601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:36.539028 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc35fd7-81c9-4b28-88c3-e683a09b7433 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:36.544907 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:36.545613 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.545893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.546165 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] PATH_INFO: `/v3/projects/0a32d58bd95b4045a564f3b3b163108c` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.547407 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a32d58bd95b4045a564f3b3b163108c)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:36.549866 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:36.591531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:36.607258 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a32d58bd95b4045a564f3b3b163108c', 'request_id': 'req-b1fff228-c263-46ea-ae8a-8c87adcca18a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.616379 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a32d58bd95b4045a564f3b3b163108c', 'request_id': 'req-b1fff228-c263-46ea-ae8a-8c87adcca18a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.623509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a32d58bd95b4045a564f3b3b163108c', 'request_id': 'req-b1fff228-c263-46ea-ae8a-8c87adcca18a'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.636389 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] The token cache is being invalidate because project 0a32d58bd95b4045a564f3b3b163108c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:19:36.636676 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.637434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-b1fff228-c263-46ea-ae8a-8c87adcca18a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:36.639139 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1418/2838] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:19:36 2022] DELETE /identity/v3/projects/0a32d58bd95b4045a564f3b3b163108c => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:36.644164 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a462027e-f3f3-4040-8e94-f35db7a2937a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.644454 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a462027e-f3f3-4040-8e94-f35db7a2937a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.644733 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a462027e-f3f3-4040-8e94-f35db7a2937a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.668425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a462027e-f3f3-4040-8e94-f35db7a2937a None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:36.669692 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a462027e-f3f3-4040-8e94-f35db7a2937a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:36.728237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c20c555-7497-48c4-9db8-9cc1402433de None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.728524 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c20c555-7497-48c4-9db8-9cc1402433de None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.728801 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c20c555-7497-48c4-9db8-9cc1402433de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.736025 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1421/2839] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:36.743558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c27df-9c7d-4ed3-b5c4-27a10c6ef31c None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.743848 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c27df-9c7d-4ed3-b5c4-27a10c6ef31c None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.744127 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c27df-9c7d-4ed3-b5c4-27a10c6ef31c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.748820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-3c20c555-7497-48c4-9db8-9cc1402433de None None] MFA Rules not processed for user `219301d335c348358d8aaa63b86ddcc9`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:36.750046 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3c20c555-7497-48c4-9db8-9cc1402433de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:36.763109 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-948c27df-9c7d-4ed3-b5c4-27a10c6ef31c None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:36.764343 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-948c27df-9c7d-4ed3-b5c4-27a10c6ef31c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:36.772197 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1422/2840] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:19:36.776257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c27df-9c7d-4ed3-b5c4-27a10c6ef31c None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:36.785252 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c27df-9c7d-4ed3-b5c4-27a10c6ef31c None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:36.789990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:36.790693 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.791286 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.791567 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.792256 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:19:36.792562 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:36.794160 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:36.810070 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1423/2841] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:19:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 26 03:19:36.810271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1419/2842] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:36 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:36.811209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:36.822377 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a39618-e150-419a-b734-1aff742bb80b admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:36.827342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:36.827959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.828235 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.828498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] PATH_INFO: `/v3/projects` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.829153 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:36.831163 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:36.866975 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1424/2843] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:19:36 2022] POST /identity/v3/projects => generated 373 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:19:36.868549 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c20c555-7497-48c4-9db8-9cc1402433de None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:36.879578 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c20c555-7497-48c4-9db8-9cc1402433de None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:36.933743 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:36.934366 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.934651 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.934930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] PATH_INFO: `/v3/users` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.935674 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:36.937603 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:36.960052 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1420/2844] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:19:36 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 26 03:19:36.964509 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:36.975246 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9209caa9-9e7d-4de5-bfd4-0bb94ebdff35 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:36.980568 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:36.981191 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:36.981474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:36.981751 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:36.982455 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:36.984210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:36.994810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1425/2845] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:19:36.999754 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:37.011190 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338669da-282d-4360-a6a8-113f6ddd3e51 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:37.016953 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183176d8-4bfd-48b7-b81c-614bc6cef7b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:37.017659 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183176d8-4bfd-48b7-b81c-614bc6cef7b8 admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:37.018006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183176d8-4bfd-48b7-b81c-614bc6cef7b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:37.018348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183176d8-4bfd-48b7-b81c-614bc6cef7b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:37.019187 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183176d8-4bfd-48b7-b81c-614bc6cef7b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:37.020863 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183176d8-4bfd-48b7-b81c-614bc6cef7b8 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:37.032100 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1421/2846] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:19:37.036244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:37.046729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec51b8d-dd7e-4c2c-9f34-d3cb829b355c admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:37.051536 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:37.052270 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:37.052547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:37.052817 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] PATH_INFO: `/v3/projects/89b7ec6e1b2048dba9cfb411ed59518a/users/3b24bfe5902b4bb6abe18ddab3e4c05b/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:37.069496 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] RBAC: Authorizing `identity:create_grant(project_id=89b7ec6e1b2048dba9cfb411ed59518a, user_id=3b24bfe5902b4bb6abe18ddab3e4c05b, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:37.072385 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:37.087591 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1426/2847] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:19:37 2022] PUT /identity/v3/projects/89b7ec6e1b2048dba9cfb411ed59518a/users/3b24bfe5902b4bb6abe18ddab3e4c05b/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:37.098614 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040d7bb1-724d-4002-966b-2a91a7da5e84 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:37.098865 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040d7bb1-724d-4002-966b-2a91a7da5e84 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:37.099344 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040d7bb1-724d-4002-966b-2a91a7da5e84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:37.128572 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-040d7bb1-724d-4002-966b-2a91a7da5e84 None None] MFA Rules not processed for user `3b24bfe5902b4bb6abe18ddab3e4c05b`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:37.129807 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-040d7bb1-724d-4002-966b-2a91a7da5e84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:37.161181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:37.199509 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1422/2848] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:37 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 26 03:19:37.210421 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:37.212341 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:37.283086 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:37.293051 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf3e7a1-a4c3-4c00-baaa-4d0a2e6c9ccf admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:37.297446 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:37.298074 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:37.298354 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:37.298618 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:37.308289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:37.309732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:37.321378 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1427/2849] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:19:37 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:19:38.749243 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9782ffff-9100-4d76-8127-ecba4b1f660c None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:38.749723 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9782ffff-9100-4d76-8127-ecba4b1f660c None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:38.750163 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9782ffff-9100-4d76-8127-ecba4b1f660c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:38.771079 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-9782ffff-9100-4d76-8127-ecba4b1f660c None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:38.771774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9782ffff-9100-4d76-8127-ecba4b1f660c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:38.839290 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1423/2850] 10.222.0.42 () {58 vars in 1009 bytes} [Tue Jul 26 03:19:38 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:38.865257 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:38.867558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:38.949077 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e777f39e-25bf-46fe-9af4-059fbefdce01 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:38.955941 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c5d4b8-98a1-478e-b692-49cd68209a7b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:38.956561 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c5d4b8-98a1-478e-b692-49cd68209a7b service swift] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:38.956804 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c5d4b8-98a1-478e-b692-49cd68209a7b service swift] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:38.957543 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c5d4b8-98a1-478e-b692-49cd68209a7b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:38.968340 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c5d4b8-98a1-478e-b692-49cd68209a7b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:38.970040 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c5d4b8-98a1-478e-b692-49cd68209a7b service swift] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:38.979630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1428/2851] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:19:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:19:47.743471 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9782ffff-9100-4d76-8127-ecba4b1f660c None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:47.814308 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9782ffff-9100-4d76-8127-ecba4b1f660c None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:47.819259 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:47.820021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:47.820416 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:47.820694 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] PATH_INFO: `/v3/users/3b24bfe5902b4bb6abe18ddab3e4c05b` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:47.821942 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b24bfe5902b4bb6abe18ddab3e4c05b)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:47.823908 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:47.874745 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b24bfe5902b4bb6abe18ddab3e4c05b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:47.885952 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b24bfe5902b4bb6abe18ddab3e4c05b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:47.893214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b24bfe5902b4bb6abe18ddab3e4c05b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:47.897030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:47.902550 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b24bfe5902b4bb6abe18ddab3e4c05b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:47.903401 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b24bfe5902b4bb6abe18ddab3e4c05b'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:47.912233 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1424/2852] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:19:47 2022] DELETE /identity/v3/users/3b24bfe5902b4bb6abe18ddab3e4c05b => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:47.916812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:47.917289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:47.917405 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:47.936489 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:19:47.937714 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:48.006339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1429/2853] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:19:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:19:48.014642 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:48.014860 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:48.081188 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033e7ed8-077e-4139-9757-b1bd2fe3feb2 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:48.084790 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:48.085423 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:48.085735 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:48.086063 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:48.095592 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:48.097038 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:48.107211 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1425/2854] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:19:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:19:48.409644 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:19:48.411157 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:19:48.424091 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92ffc77-3852-4f89-bd8a-a7bff14420fd None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:19:48.428769 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:19:48.429425 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:19:48.429701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:19:48.429982 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] PATH_INFO: `/v3/projects/89b7ec6e1b2048dba9cfb411ed59518a` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:19:48.431159 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=89b7ec6e1b2048dba9cfb411ed59518a)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:19:48.433112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:19:48.482768 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89b7ec6e1b2048dba9cfb411ed59518a', 'request_id': 'req-42dd8cf1-5282-49f4-85cf-59c527af0f54'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:48.491574 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89b7ec6e1b2048dba9cfb411ed59518a', 'request_id': 'req-42dd8cf1-5282-49f4-85cf-59c527af0f54'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:48.497066 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89b7ec6e1b2048dba9cfb411ed59518a', 'request_id': 'req-42dd8cf1-5282-49f4-85cf-59c527af0f54'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:48.504059 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:48.509278 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] The token cache is being invalidate because project 89b7ec6e1b2048dba9cfb411ed59518a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80428) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:19:48.509666 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:48.510450 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:19:48.512112 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1430/2855] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:19:48 2022] DELETE /identity/v3/projects/89b7ec6e1b2048dba9cfb411ed59518a => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:19:58.102715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=28 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:19:58.518740 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:20:04.710365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:20:04.719817 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499ba1b8-4470-49cd-81e9-ba512afeb98f service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:20:04.773066 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:20:04.773683 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:20:04.773959 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:20:04.774238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:20:04.776347 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:20:04.843585 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:20:04.845033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:20:04.854803 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1426/2856] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:20:04 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:20:14.850647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=50 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:20:35.010600 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:20:35.012295 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:20:35.026083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dd8cf1-5282-49f4-85cf-59c527af0f54 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:20:35.030598 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:20:35.031233 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:20:35.031487 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:20:35.031759 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:20:35.033715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:20:35.117637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:20:35.119120 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:20:35.139220 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1431/2857] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:20:35 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:20:36.339502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:20:36.341181 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:20:36.352892 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abf63ba-9958-4c3f-9c44-6e7de6a94195 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:20:36.357466 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:20:36.358085 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:20:36.358365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:20:36.358640 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:20:36.360819 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:20:36.375097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:20:36.376529 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:20:36.387749 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1427/2858] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:20:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:20:45.136762 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=43 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:20:46.382915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:22.162244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:22.173498 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d386f-6589-438a-be83-87ded634ccc2 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:22.177730 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eb78ad-47a5-4539-9d04-f91cec175418 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:22.177992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb78ad-47a5-4539-9d04-f91cec175418 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:22.178260 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb78ad-47a5-4539-9d04-f91cec175418 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:22.178531 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb78ad-47a5-4539-9d04-f91cec175418 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:22.180371 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-30eb78ad-47a5-4539-9d04-f91cec175418 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:22.192079 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eb78ad-47a5-4539-9d04-f91cec175418 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:22.193538 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eb78ad-47a5-4539-9d04-f91cec175418 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:22.211581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1432/2859] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:21:22 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:21:28.237474 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:28.239919 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:28.255780 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc3265f-a7b2-4441-8622-85f8c35be86a service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:28.261968 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:28.263005 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:28.263595 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:28.264033 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:28.266006 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:28.279543 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:28.281132 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:28.291970 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1428/2860] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:21:28 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:21:29.384085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:29.384573 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:29.384676 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:29.437749 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] MFA Rules not processed for user `b211db19066a4767b09f1fcae194085a`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:21:29.438978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:29.517216 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1433/2861] 10.222.0.42 () {58 vars in 967 bytes} [Tue Jul 26 03:21:29 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:21:29.558278 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:29.559820 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:29.652873 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416204bb-8713-4087-87f5-0cb371ae471a service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:29.662463 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'afebddca912546119eea61f568bbd60d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:29.663378 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:29.663726 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:29.664016 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:29.677690 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:29.679338 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:29.688858 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1429/2862] 10.222.0.42 () {60 vars in 1384 bytes} [Tue Jul 26 03:21:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 26 03:21:36.404533 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:36.405995 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:36.475508 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73dcef7-c8ca-43f8-a93a-ee9234f7a510 None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:36.481296 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:36.482189 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:36.482463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:36.482732 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] PATH_INFO: `/v3/users/219301d335c348358d8aaa63b86ddcc9` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:36.483960 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=219301d335c348358d8aaa63b86ddcc9)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:36.485869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:36.537221 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '219301d335c348358d8aaa63b86ddcc9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.544339 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '219301d335c348358d8aaa63b86ddcc9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.547987 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:36.555928 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '219301d335c348358d8aaa63b86ddcc9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.569835 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '219301d335c348358d8aaa63b86ddcc9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.570462 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '219301d335c348358d8aaa63b86ddcc9'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.576376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1434/2863] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:21:36 2022] DELETE /identity/v3/users/219301d335c348358d8aaa63b86ddcc9 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:21:36.768222 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:36.784319 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4676b27-60b9-41be-8306-8bbfca57eb9c service swift] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:36.844294 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:36.844941 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:36.845219 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:36.845495 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] PATH_INFO: `/v3/projects/ac6c4069e73d4966aa373c5cc84d8c00` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:36.846682 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac6c4069e73d4966aa373c5cc84d8c00)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:36.848634 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:36.895220 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac6c4069e73d4966aa373c5cc84d8c00', 'request_id': 'req-35e1ea93-8733-4541-b601-ffb48a6cec1c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.903590 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac6c4069e73d4966aa373c5cc84d8c00', 'request_id': 'req-35e1ea93-8733-4541-b601-ffb48a6cec1c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.905616 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:36.910322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac6c4069e73d4966aa373c5cc84d8c00', 'request_id': 'req-35e1ea93-8733-4541-b601-ffb48a6cec1c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.922255 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] The token cache is being invalidate because project ac6c4069e73d4966aa373c5cc84d8c00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:21:36.922545 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.923200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:36.924775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1430/2864] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:21:36 2022] DELETE /identity/v3/projects/ac6c4069e73d4966aa373c5cc84d8c00 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:21:36.929076 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:36.930698 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:36.944978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3cc0c6-065f-4654-8775-1996db813b23 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.002977 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.003670 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.003950 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.004224 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] PATH_INFO: `/v3/users/61b086c34a114c06b35bd2b54a55014d` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.005439 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] RBAC: Authorizing `identity:delete_user(user_id=61b086c34a114c06b35bd2b54a55014d)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.007427 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.054838 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61b086c34a114c06b35bd2b54a55014d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.061585 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61b086c34a114c06b35bd2b54a55014d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.070328 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61b086c34a114c06b35bd2b54a55014d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.085905 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61b086c34a114c06b35bd2b54a55014d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.086631 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-0461b9fc-2798-4e78-930a-de6b0e8cf659 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61b086c34a114c06b35bd2b54a55014d'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.092901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1435/2865] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:21:36 2022] DELETE /identity/v3/users/61b086c34a114c06b35bd2b54a55014d => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:21:37.278870 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:37.280376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:37.296177 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e1ea93-8733-4541-b601-ffb48a6cec1c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.357449 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.358138 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.358417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.358693 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] PATH_INFO: `/v3/projects/c5fdb1d2ed3c4ab4977d81294e5d821f` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.359930 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5fdb1d2ed3c4ab4977d81294e5d821f)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.361933 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.411974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5fdb1d2ed3c4ab4977d81294e5d821f', 'request_id': 'req-529966ec-c8bc-4328-b23a-0b92bdab1e0c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.421437 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5fdb1d2ed3c4ab4977d81294e5d821f', 'request_id': 'req-529966ec-c8bc-4328-b23a-0b92bdab1e0c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.428097 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5fdb1d2ed3c4ab4977d81294e5d821f', 'request_id': 'req-529966ec-c8bc-4328-b23a-0b92bdab1e0c'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.439915 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] The token cache is being invalidate because project c5fdb1d2ed3c4ab4977d81294e5d821f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:21:37.440200 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.441291 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-529966ec-c8bc-4328-b23a-0b92bdab1e0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:21:37.443254 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1431/2866] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:21:37 2022] DELETE /identity/v3/projects/c5fdb1d2ed3c4ab4977d81294e5d821f => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:21:37.448486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5c7144-4415-4b55-aea4-816dc72bed8d None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.448953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5c7144-4415-4b55-aea4-816dc72bed8d None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.449061 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5c7144-4415-4b55-aea4-816dc72bed8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.468896 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-ad5c7144-4415-4b55-aea4-816dc72bed8d None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:21:37.470101 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ad5c7144-4415-4b55-aea4-816dc72bed8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:37.521205 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:37.537818 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1436/2867] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:21:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:21:37.542101 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c4078-14f8-4917-86f9-9d59cc38d5af None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.542420 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c4078-14f8-4917-86f9-9d59cc38d5af None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.542666 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894c4078-14f8-4917-86f9-9d59cc38d5af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.564501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-894c4078-14f8-4917-86f9-9d59cc38d5af None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:21:37.565643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-894c4078-14f8-4917-86f9-9d59cc38d5af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:37.574506 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1432/2868] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:21:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:21:37.578992 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5c7144-4415-4b55-aea4-816dc72bed8d None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:37.590580 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5c7144-4415-4b55-aea4-816dc72bed8d None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.595881 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.596512 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.596807 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.597210 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] PATH_INFO: `/v3/domains` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.597917 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80428) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 26 03:21:37.598259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.600056 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.612285 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1437/2869] 10.222.0.42 () {58 vars in 1153 bytes} [Tue Jul 26 03:21:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 26 03:21:37.616779 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894c4078-14f8-4917-86f9-9d59cc38d5af None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:37.626118 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894c4078-14f8-4917-86f9-9d59cc38d5af None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.630943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.631691 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.631967 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.632238 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.632902 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.634800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.678482 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1433/2870] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:21:37 2022] POST /identity/v3/projects => generated 379 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:21:37.684920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:37.700855 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831ab3e6-ee05-4b70-a0c1-f9937195f01d admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.761140 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.761748 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.762011 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.762283 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.762984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.764901 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.787847 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1438/2871] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:21:37 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:21:37.792096 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:37.802618 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f47841-6cd1-4513-aa98-69a236a8d956 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.807468 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.808087 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.808370 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.808645 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.809365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.810991 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.821628 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1434/2872] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:21:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:21:37.825560 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:37.835673 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27872d66-8b02-4764-a502-6eeb09fe147f admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.840356 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffd2e66-90a1-4a12-be59-3057fca647a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.840959 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd2e66-90a1-4a12-be59-3057fca647a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.841234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd2e66-90a1-4a12-be59-3057fca647a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.841503 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffd2e66-90a1-4a12-be59-3057fca647a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.842192 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffd2e66-90a1-4a12-be59-3057fca647a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.843853 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffd2e66-90a1-4a12-be59-3057fca647a3 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.855030 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1439/2873] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:21:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:21:37.859329 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:37.869925 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb32806-4587-48df-9ed0-ea00752d889a admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:37.875501 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:37.876157 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.876434 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.876706 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] PATH_INFO: `/v3/projects/d582e70969cb448b86cd6c9ca31512ee/users/0b79af958ad24a8e91fc18110526e2d7/roles/d0d63bdb855c485cabc4d6711c761e39` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.892601 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d582e70969cb448b86cd6c9ca31512ee, user_id=0b79af958ad24a8e91fc18110526e2d7, role_id=d0d63bdb855c485cabc4d6711c761e39)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:37.895266 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:37.912171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1435/2874] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:21:37 2022] PUT /identity/v3/projects/d582e70969cb448b86cd6c9ca31512ee/users/0b79af958ad24a8e91fc18110526e2d7/roles/d0d63bdb855c485cabc4d6711c761e39 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:21:37.917361 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5728b4d-4517-400d-8c10-c75f705face8 None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:37.917648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5728b4d-4517-400d-8c10-c75f705face8 None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:37.917932 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5728b4d-4517-400d-8c10-c75f705face8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:37.937766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-a5728b4d-4517-400d-8c10-c75f705face8 None None] MFA Rules not processed for user `84df6af7bcfb4cfca1045798a2ef6d69`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:21:37.939009 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-a5728b4d-4517-400d-8c10-c75f705face8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:37.999859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1440/2875] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:21:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:21:38.009686 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:38.011465 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:38.018356 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=89,keystone:DELETE=8,keystone:INSERT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:38.070282 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c700c35-8782-4207-ab04-7490697e6bc0 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:38.074460 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:38.075062 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:38.075382 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:38.075653 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:38.085507 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:38.086931 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:38.097372 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1436/2876] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:21:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:21:42.273402 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c82d57-77ad-476f-a097-ae69adce3a2a None None] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:42.274237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c82d57-77ad-476f-a097-ae69adce3a2a None None] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:42.274701 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c82d57-77ad-476f-a097-ae69adce3a2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:42.304100 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.auth.core [None req-36c82d57-77ad-476f-a097-ae69adce3a2a None None] MFA Rules not processed for user `0b79af958ad24a8e91fc18110526e2d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80428) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:21:42.305342 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-36c82d57-77ad-476f-a097-ae69adce3a2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:42.365336 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:42.377869 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1441/2877] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:21:42 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:21:42.382001 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:42.392348 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d51f259-c2d3-4ed5-b51c-e77da48457e5 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:42.397113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:42.397748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:42.398030 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:42.398311 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] PATH_INFO: `/v3/projects` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:42.399037 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:42.400992 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:42.430531 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1437/2878] 10.222.0.42 () {60 vars in 1153 bytes} [Tue Jul 26 03:21:42 2022] POST /identity/v3/projects => generated 377 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:21:42.434584 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c82d57-77ad-476f-a097-ae69adce3a2a None None] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:42.446276 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c82d57-77ad-476f-a097-ae69adce3a2a None None] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:42.505974 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:42.506805 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] REQUEST_METHOD: `POST` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:42.507073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:42.507345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] PATH_INFO: `/v3/users` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:42.508020 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:42.509986 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:42.536742 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1442/2879] 10.222.0.42 () {60 vars in 1144 bytes} [Tue Jul 26 03:21:42 2022] POST /identity/v3/users => generated 295 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 26 03:21:42.540951 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:42.551436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f9e50c-9b50-4753-b7f7-ef817328e784 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:42.556158 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:42.556774 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:42.557032 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:42.557337 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:42.558009 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:42.559688 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:42.571561 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1438/2880] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:21:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:21:42.577394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:42.591892 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9a6f8b-91ca-40b9-92a1-e254fa15fc43 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:42.600444 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:42.601110 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:42.601383 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:42.601651 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] PATH_INFO: `/v3/projects/ef345f35153a4a2d8512c25864316b63/users/90eff395b696497f9706e1ddcb8d839a/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:42.616708 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef345f35153a4a2d8512c25864316b63, user_id=90eff395b696497f9706e1ddcb8d839a, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:42.620073 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:42.635534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1443/2881] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:21:42 2022] PUT /identity/v3/projects/ef345f35153a4a2d8512c25864316b63/users/90eff395b696497f9706e1ddcb8d839a/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:21:42.639500 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:42.649773 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ec96e5-ed5a-4a92-894d-2670a7b0031c admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:42.707635 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac98103d-8290-487f-b53a-e6a3002bae6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:42.708271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98103d-8290-487f-b53a-e6a3002bae6f admin admin] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:42.708573 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98103d-8290-487f-b53a-e6a3002bae6f admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:42.708849 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac98103d-8290-487f-b53a-e6a3002bae6f admin admin] PATH_INFO: `/v3/roles` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:42.709541 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac98103d-8290-487f-b53a-e6a3002bae6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:42.711178 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac98103d-8290-487f-b53a-e6a3002bae6f admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:42.721851 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1439/2882] 10.222.0.42 () {58 vars in 1122 bytes} [Tue Jul 26 03:21:42 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 26 03:21:42.726237 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:42.736921 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8cfede-d7db-4aad-ad1e-6df731a307c4 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:42.742603 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:42.743309 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] REQUEST_METHOD: `PUT` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:42.743583 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:42.743834 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] PATH_INFO: `/v3/projects/ef345f35153a4a2d8512c25864316b63/users/90eff395b696497f9706e1ddcb8d839a/roles/50e7a7f2795f407e8da49801c56f6c5b` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:42.746035 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef345f35153a4a2d8512c25864316b63, user_id=90eff395b696497f9706e1ddcb8d839a, role_id=50e7a7f2795f407e8da49801c56f6c5b)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:42.748827 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:42.761422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1444/2883] 10.222.0.42 () {60 vars in 1483 bytes} [Tue Jul 26 03:21:42 2022] PUT /identity/v3/projects/ef345f35153a4a2d8512c25864316b63/users/90eff395b696497f9706e1ddcb8d839a/roles/50e7a7f2795f407e8da49801c56f6c5b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:21:47.131113 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e175184-2a75-41ab-8f2d-33be5612fc91 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:47.131629 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e175184-2a75-41ab-8f2d-33be5612fc91 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:47.131736 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e175184-2a75-41ab-8f2d-33be5612fc91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:47.174296 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-6e175184-2a75-41ab-8f2d-33be5612fc91 None None] MFA Rules not processed for user `90eff395b696497f9706e1ddcb8d839a`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:21:47.177007 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-6e175184-2a75-41ab-8f2d-33be5612fc91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:47.267974 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:47.310236 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1440/2884] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:21:47 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:21:47.320652 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:21:47.322963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:21:47.413161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce57fe5f-3031-4930-9dc5-9310b7afe877 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:21:47.418793 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:21:47.419472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:21:47.419752 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:21:47.420027 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:21:47.472298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:47.491394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:21:47.493015 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:21:47.502510 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1445/2885] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:21:47 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:21:57.287672 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=2 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:21:57.498259 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=8 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:22:06.153588 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:22:06.154122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:22:06.154122 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:22:06.174804 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] MFA Rules not processed for user `0b79af958ad24a8e91fc18110526e2d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:22:06.176020 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:22:06.183580 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1441/2886] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:22:06 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:22:06.513448 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:22:06.515172 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:22:06.575990 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b20230-a989-49df-bbd1-0c1d9eac2024 service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:22:06.580161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:22:06.580786 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:22:06.581062 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:22:06.581334 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:22:06.583500 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:22:06.593313 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:22:06.594747 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:22:06.602991 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1446/2887] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:22:06 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:22:08.089275 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:22:08.090843 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:22:08.159566 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f71e03-12a9-4c62-9472-8487f9ec4b0d None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:22:08.164647 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:22:08.165387 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:22:08.165669 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:22:08.165944 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:22:08.167748 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:22:08.179056 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:22:08.180636 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:22:08.190800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1442/2888] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:22:08 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:22:16.599710 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=26 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:22:18.186488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=35 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:22:21.540984 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:22:21.543266 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:22:21.560620 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8f9c9-d11d-4f2d-8cf4-0ebe1a694bb2 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:22:21.567586 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:22:21.568209 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:22:21.568486 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:22:21.568756 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:22:21.570677 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:22:21.583699 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:22:21.585293 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:22:21.596182 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1447/2889] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:22:21 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:22:24.868376 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:22:24.877932 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e239da69-6a97-44ce-9725-295b7a1d4799 service placement] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:22:24.881760 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfa5901-bd5f-4164-bc4a-2e9a38191f4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:22:24.882365 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfa5901-bd5f-4164-bc4a-2e9a38191f4f service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:22:24.882643 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfa5901-bd5f-4164-bc4a-2e9a38191f4f service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:22:24.882912 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfa5901-bd5f-4164-bc4a-2e9a38191f4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:22:24.892390 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfa5901-bd5f-4164-bc4a-2e9a38191f4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:22:24.893813 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfa5901-bd5f-4164-bc4a-2e9a38191f4f service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:22:24.903470 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1443/2890] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:22:24 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:22:31.591547 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=6 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:22:34.899164 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:23:54.678311 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:23:54.680032 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:23:54.695844 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a4522f-8f5c-4d95-bb04-dfa9afa550d6 service nova] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:23:54.703492 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:23:54.704105 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:23:54.704379 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:23:54.704648 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:23:54.706715 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:23:54.718628 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:23:54.720244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:23:54.730083 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1448/2891] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:23:54 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:24:04.726130 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=6 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:24:26.501171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e616ed-5158-4674-a0dd-c0f0b9079316 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:24:26.501719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e616ed-5158-4674-a0dd-c0f0b9079316 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:24:26.501719 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e616ed-5158-4674-a0dd-c0f0b9079316 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:24:26.522237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-d1e616ed-5158-4674-a0dd-c0f0b9079316 None None] MFA Rules not processed for user `0b79af958ad24a8e91fc18110526e2d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:24:26.523505 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-d1e616ed-5158-4674-a0dd-c0f0b9079316 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:24:26.531831 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1444/2892] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:24:26 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:24:26.848067 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:24:26.858472 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b09144-99e9-4082-9be0-80979ffae407 service placement] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:24:26.862442 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:24:26.863085 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:24:26.863376 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:24:26.863647 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:24:26.865551 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:24:26.875810 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:24:26.877394 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:24:26.885859 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1449/2893] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:24:26 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:24:36.517248 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=3 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:24:36.882169 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=6 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:26:18.858800 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5e879-81a2-4296-85c7-5b473714c1a4 None None] REQUEST_METHOD: `POST` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:26:18.859425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5e879-81a2-4296-85c7-5b473714c1a4 None None] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:26:18.859425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5e879-81a2-4296-85c7-5b473714c1a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:26:18.888454 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.auth.core [None req-87d5e879-81a2-4296-85c7-5b473714c1a4 None None] MFA Rules not processed for user `0b79af958ad24a8e91fc18110526e2d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80427) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 26 03:26:18.889649 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-87d5e879-81a2-4296-85c7-5b473714c1a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:26:18.897408 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1445/2894] 10.222.0.42 () {58 vars in 958 bytes} [Tue Jul 26 03:26:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 26 03:26:18.938102 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:26:18.939812 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:26:19.001997 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099877ee-f19a-438f-9976-b02eac2faead service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:26:19.008587 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '48f06347846c48c2933cee7ebd32b947', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:26:19.009223 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:26:19.009501 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:26:19.009772 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:26:19.011980 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:26:19.028422 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:26:19.029953 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:26:19.041850 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1450/2895] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:26:18 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:26:19.818971 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d5e879-81a2-4296-85c7-5b473714c1a4 None None] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:26:19.829346 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d5e879-81a2-4296-85c7-5b473714c1a4 None None] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:26:19.833599 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:26:19.834214 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:26:19.834494 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:26:19.834884 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:26:19.837110 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:26:19.850934 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:26:19.852409 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:26:19.864313 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1446/2896] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:26:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:26:21.632463 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:26:21.643706 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d48d26-83b3-4ab0-9027-8631d179e270 service cinder] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:26:21.647595 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:26:21.648206 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:26:21.648451 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:26:21.648750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:26:21.650689 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:26:21.660474 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:26:21.661900 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:26:21.670529 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1451/2897] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:26:21 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 26 03:26:29.859156 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=9 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:26:31.666308 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=32 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:26:37.667900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:26:37.679461 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4bfa5e-f350-40b4-ae00-e501adedcc92 service neutron] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:26:37.684893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:26:37.685909 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:26:37.686171 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:26:37.686453 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:26:37.688237 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:26:37.700725 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:26:37.702322 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:26:37.713412 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1447/2898] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:26:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:26:47.708896 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:28:16.649741 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:16.660875 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0d2f2a-786b-4a80-a5d1-603d831e53cd service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:16.666161 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:16.666873 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:16.667234 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:16.667557 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:16.677796 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:16.679269 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:16.691345 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1452/2899] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:28:16 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:28:19.962246 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:19.976510 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcea5dd-5a3a-491e-be9c-db32082fdb3b service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:19.982900 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e2c09978fa3f44d895cbc17efdca2b32', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:19.983552 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:19.983829 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:19.984099 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:19.986059 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:28:20.000943 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:20.002846 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:20.016170 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1448/2900] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:28:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:28:26.684566 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=6 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:28:29.938860 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:29.948298 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a0965f-d1f3-4ab7-9146-181555110c87 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:29.969348 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '29a1868eb95f4396aa3fd2da677b40f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:29.969973 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:29.970253 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:29.970534 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:30.011011 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=6 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:28:30.037750 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:30.039240 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:30.049129 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1453/2901] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:28:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:28:32.860557 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:32.893287 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1481a60b-41cd-4171-9a46-c5dd1edd9648 service nova] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:32.899770 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:32.900417 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] REQUEST_METHOD: `GET` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:32.900696 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:32.901105 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:32.902893 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:28:32.916436 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:32.918082 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:32.934502 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1449/2902] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:28:32 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:28:36.146426 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:36.206731 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe33be-a5cf-4d5b-acda-d798a63b83d2 service neutron] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:36.211362 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:36.212008 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:36.212289 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:36.212564 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] PATH_INFO: `/v3/users/0b79af958ad24a8e91fc18110526e2d7` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:36.213722 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b79af958ad24a8e91fc18110526e2d7)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:36.215920 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:36.268856 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b79af958ad24a8e91fc18110526e2d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.275978 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b79af958ad24a8e91fc18110526e2d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.285400 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b79af958ad24a8e91fc18110526e2d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.297165 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b79af958ad24a8e91fc18110526e2d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.297780 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b79af958ad24a8e91fc18110526e2d7'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.303630 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1454/2903] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:28:36 2022] DELETE /identity/v3/users/0b79af958ad24a8e91fc18110526e2d7 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 26 03:28:36.482625 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:36.519285 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8638e8-c5a3-4f53-8dbf-19602e0b0729 service placement] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:36.575271 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:36.575966 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:36.576247 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:36.576521 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] PATH_INFO: `/v3/projects/d582e70969cb448b86cd6c9ca31512ee` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:36.577705 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d582e70969cb448b86cd6c9ca31512ee)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:36.579670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:36.626425 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd582e70969cb448b86cd6c9ca31512ee', 'request_id': 'req-21453e89-bd64-4dea-ab67-eb6695b91ae5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.634823 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd582e70969cb448b86cd6c9ca31512ee', 'request_id': 'req-21453e89-bd64-4dea-ab67-eb6695b91ae5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.640297 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd582e70969cb448b86cd6c9ca31512ee', 'request_id': 'req-21453e89-bd64-4dea-ab67-eb6695b91ae5'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.664207 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] The token cache is being invalidate because project d582e70969cb448b86cd6c9ca31512ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:28:36.664477 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.665228 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.666854 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1450/2904] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:28:36 2022] DELETE /identity/v3/projects/d582e70969cb448b86cd6c9ca31512ee => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:28:36.671481 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:36.673138 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:28:36.698186 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22c4503-ec3a-44b0-8616-c114cc8681e0 admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:36.711244 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:28:36.766222 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:36.766867 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] REQUEST_METHOD: `DELETE` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:36.767170 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:36.767558 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] PATH_INFO: `/v3/users/90eff395b696497f9706e1ddcb8d839a` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:36.768792 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] RBAC: Authorizing `identity:delete_user(user_id=90eff395b696497f9706e1ddcb8d839a)` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:36.770729 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:36.819800 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '90eff395b696497f9706e1ddcb8d839a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.826590 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '90eff395b696497f9706e1ddcb8d839a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.834688 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '90eff395b696497f9706e1ddcb8d839a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.847145 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '90eff395b696497f9706e1ddcb8d839a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.847784 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.notifications [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '90eff395b696497f9706e1ddcb8d839a'} {{(pid=80428) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:36.853766 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1455/2905] 10.222.0.42 () {58 vars in 1224 bytes} [Tue Jul 26 03:28:36 2022] DELETE /identity/v3/users/90eff395b696497f9706e1ddcb8d839a => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:28:37.034398 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Authenticating user token {{(pid=80427) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:28:37.036142 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.fernet_utils [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80427) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:28:37.051715 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21453e89-bd64-4dea-ab67-eb6695b91ae5 admin admin] Validating token access rules against request {{(pid=80427) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:28:37.095587 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:28:37.113582 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '84df6af7bcfb4cfca1045798a2ef6d69', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b8c4cb2e7662425d87dcbcca96df5974', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80427) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:28:37.114227 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:28:37.114504 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:28:37.114775 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] PATH_INFO: `/v3/projects/ef345f35153a4a2d8512c25864316b63` {{(pid=80427) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:28:37.115984 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef345f35153a4a2d8512c25864316b63)` {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:28:37.117901 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] RBAC: Authorization granted {{(pid=80427) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:28:37.166021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef345f35153a4a2d8512c25864316b63', 'request_id': 'req-aaf92853-86e7-42a8-bf70-3cb8258caf12'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:37.180670 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef345f35153a4a2d8512c25864316b63', 'request_id': 'req-aaf92853-86e7-42a8-bf70-3cb8258caf12'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:37.189021 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef345f35153a4a2d8512c25864316b63', 'request_id': 'req-aaf92853-86e7-42a8-bf70-3cb8258caf12'} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:37.201198 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] The token cache is being invalidate because project ef345f35153a4a2d8512c25864316b63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80427) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 26 03:28:37.201488 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:37.202205 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG keystone.notifications [None req-aaf92853-86e7-42a8-bf70-3cb8258caf12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80427) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 26 03:28:37.203886 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: [pid: 80427|app: 0|req: 1451/2906] 10.222.0.42 () {58 vars in 1233 bytes} [Tue Jul 26 03:28:37 2022] DELETE /identity/v3/projects/ef345f35153a4a2d8512c25864316b63 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 26 03:28:46.850428 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=45,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:28:47.197729 dv-cinder-850840-2-iscsi devstack@keystone.service[80427]: DEBUG dbcounter [-] [80427] Writing DB stats keystone:SELECT=32,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80427) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 26 03:29:04.607292 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Authenticating user token {{(pid=80428) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 26 03:29:04.608963 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:29:04.721822 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8b3d1a-af97-4546-872f-2352d42024db admin admin] Validating token access rules against request {{(pid=80428) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 26 03:29:04.794431 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a1f061-a199-4554-93ca-584f55c73a4c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35f28f9458e54fb8a76c740d6f9c33bf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e3a97d61573e450fa7ad485016b1f019', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80428) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 26 03:29:04.795084 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1f061-a199-4554-93ca-584f55c73a4c service placement] REQUEST_METHOD: `GET` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 26 03:29:04.795365 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1f061-a199-4554-93ca-584f55c73a4c service placement] SCRIPT_NAME: `/identity` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 26 03:29:04.795637 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1f061-a199-4554-93ca-584f55c73a4c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80428) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 26 03:29:04.797619 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.fernet_utils [None req-05a1f061-a199-4554-93ca-584f55c73a4c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80428) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 26 03:29:04.810915 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a1f061-a199-4554-93ca-584f55c73a4c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 26 03:29:04.812581 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a1f061-a199-4554-93ca-584f55c73a4c service placement] RBAC: Authorization granted {{(pid=80428) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 26 03:29:04.823104 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: [pid: 80428|app: 0|req: 1456/2907] 10.222.0.42 () {60 vars in 1365 bytes} [Tue Jul 26 03:29:04 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 26 03:29:14.818888 dv-cinder-850840-2-iscsi devstack@keystone.service[80428]: DEBUG dbcounter [-] [80428] Writing DB stats keystone:SELECT=74 {{(pid=80428) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}